I would like to rephrase the "Description of Problem" again:
please attach vdsm.log and supervdsm.log after the upgrade
I have asked custo(In reply to Yaniv Bronhaim from comment #4) > please attach vdsm.log and supervdsm.log after the upgrade I have asked customer for supervdsm.log file as it was not captured in Logcollector. From available logs: ---->o--->o--- 2013-11-20 05:55:04,205 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-72) [6c590a72] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 4e9a4c17-3306-4fff-a2cf-260 823c135c0 Type: VDS 2013-11-20 05:55:04,215 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [6c590a72] START, SetVdsStatusVDSCommand(HostName = lome.brd-klee.dk, HostId = 4e9a4c17-3306-4fff-a2cf-260823c135c0, status =NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6173dbb2 Thread-15917162::INFO::2013-11-20 04:55:40,438::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) Thread-15917162::DEBUG::2013-11-20 04:55:40,439::task::495::TaskManager.Task::(_debug) Task 75a79a93-a4b5-48cc-8b88-e9841d1af799: moving from state init -> state preparing Thread-15917162::DEBUG::2013-11-20 04:55:40,442::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`83437234-979f-4217-932a-a16c18370f3d`::Request was made in '/usr/share /vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' Thread-15917162::DEBUG::2013-11-20 04:55:40,443::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' Thread-15917162::DEBUG::2013-11-20 04:55:40,446::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) Thread-15917162::DEBUG::2013-11-20 04:55:40,447::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`83437234-979f-4217-932a-a16c18370f3d`::Granted request Thread-15917162::DEBUG::2013-11-20 04:55:40,449::task::495::TaskManager.Task::(_debug) Task 75a79a93-a4b5-48cc-8b88-e9841d1af799: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) Thread-15917162::DEBUG::2013-11-20 04:55:40,450::task::495::TaskManager.Task::(_debug) Task 75a79a93-a4b5-48cc-8b88-e9841d1af799: ref 1 aborting False Thread-15917162::WARNING::2013-11-20 04:55:40,463::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable Traceback (most recent call last): File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo File "/usr/share/vdsm/storage/sd.py", line 663, in isISO File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ KeyError: 'CLASS' Thread-15917162::DEBUG::2013-11-20 04:55:40,469::task::495::TaskManager.Task::(_debug) Task 75a79a93-a4b5-48cc-8b88-e9841d1af799: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', ' version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'iso prefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} [pbandark@dhcp223-129 tunis-2013112005001384923603]$ cat sos_commands/vdsm/su_vdsm_-s_.usr.bin.python_.usr.share.vdsm.dumpStorageTable.pyc Traceback (most recent call last): File "/usr/share/vdsm/dumpStorageTable.py", line 278, in <module> File "/usr/share/vdsm/dumpStorageTable.py", line 13, in __init__ File "/usr/share/vdsm/dumpStorageTable.py", line 153, in _buildStorageTable File "/usr/share/vdsm/dumpStorageTable.py", line 56, in _getSds KeyError: 'imageslist' engine=# select id,name,compatibility_version from storage_pool; id | name | compatibility_version --------------------------------------+---------+----------------------- c4e5889a-348d-11e2-83fe-9f72369eab30 | Default | 3.0 (1 row) --o<----o<--- Might be some metadata corruption issue? To confirm I have also asked to provide ISO & export metadata files.
Created attachment 827386 [details] vdsm.log
Created attachment 829107 [details] iso-metadata
Created attachment 829108 [details] export-metadata
Customer do not find supervdsm.log file in /var/log/vdsm/ directory. I have attached iso-metadata and export-metadata file provided by the customer.
Yaniv, Please check the data provided until now and let me know if we need anything else. Thanks
Fede, is this related to the pool upgrade? Seems like we changed the format there? Thread-15917162::WARNING::2013-11-20 04:55:40,463::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable Traceback (most recent call last): File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo File "/usr/share/vdsm/storage/sd.py", line 663, in isISO File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ KeyError: 'CLASS'
(In reply to Allon Mureinik from comment #11) > Fede, is this related to the pool upgrade? Seems like we changed the format > there? > > Thread-15917162::WARNING::2013-11-20 > 04:55:40,463::sp::1065::Storage.StoragePool::(getInfo) Could not get full > domain information, it is probably unavailable > Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ > KeyError: 'CLASS' The domain metadata is not changed during the pool upgrade. Looking at the two metadata attached I can see the CLASS entry. Is there any other domain in the pool?
Customer has 4 storage domains: 45cdff8b-89c0-4926-a8c7-80354064bfa6 - ISO which metadata is attached to this case. We can see that it contains the CLASS entry. 5d7b5178-3790-472c-a13b-b481eb27a61e - Export, its metadata is attached and it contains the CLASS entry either. What about these? 6b892067-ac7e-4039-a7be-5b7387525b16 772472cb-2429-466d-96cd-00b3301ef78a Can you, please, check?
(In reply to Sergey Gotliv from comment #13) > Customer has 4 storage domains: > > 45cdff8b-89c0-4926-a8c7-80354064bfa6 - ISO which metadata is attached to > this case. We can see that it contains the CLASS entry. > > 5d7b5178-3790-472c-a13b-b481eb27a61e - Export, its metadata is attached and > it contains the CLASS entry either. > > What about these? > > 6b892067-ac7e-4039-a7be-5b7387525b16 > > 772472cb-2429-466d-96cd-00b3301ef78a > > Can you, please, check? 6b892067-ac7e-4039-a7be-5b7387525b16 772472cb-2429-466d-96cd-00b3301ef78a Storage type for above domains is FC. I think you are looking for storage domain metadata. If I am correct then, here is the file: --- https://10.65.231.4/sysreports/1641939/00985286/log-collector-data/192.168.100.133/tunis-2013112005001384923603/sos_commands/vdsm/lvm_pvs_-v_-o_all ---
--sorry about the flag change, setting it back--
ok - let me check with them.
Created attachment 832144 [details] List of packages used in rhevh-6.3-20121012.0 vdsm related versions are: vdsm-4.9-113.4.el6_3.x86_64 (RSA/8, Fri Oct 5 15:11:43 2012, Key ID 199e2f91fd431d51) vdsm-cli-4.9-113.4.el6_3.x86_64 (RSA/8, Fri Oct 5 15:11:45 2012, Key ID 199e2f91fd431d51) vdsm-hook-vhostmd-4.9-113.4.el6_3.x86_64 (RSA/8, Fri Oct 5 15:11:46 2012, Key ID 199e2f91fd431d51) vdsm-reg-4.9-113.4.el6_3.x86_64 (RSA/8, Fri Oct 5 15:11:47 2012, Key ID 199e2f91fd431d51)
*** This bug has been marked as a duplicate of bug 1031634 ***