Bug 1032598 - Unable to activate 6.3 RHEV-Hypervisor after upgrading to RHEV-Manager 3.2
Unable to activate 6.3 RHEV-Hypervisor after upgrading to RHEV-Manager 3.2
Status: CLOSED DUPLICATE of bug 1031634
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine (Show other bugs)
3.2.0
All Linux
urgent Severity high
: ---
: 3.4.0
Assigned To: Sergey Gotliv
Aharon Canan
storage
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-11-20 07:56 EST by Anand Nande
Modified: 2016-02-10 13:15 EST (History)
18 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-12-11 10:16:26 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Storage
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
vdsm.log (18.64 MB, text/plain)
2013-11-21 12:20 EST, Pratik Pravin Bandarkar
no flags Details
iso-metadata (338 bytes, text/plain)
2013-11-26 03:41 EST, Nikhil Gupta
no flags Details
export-metadata (331 bytes, text/plain)
2013-11-26 03:41 EST, Nikhil Gupta
no flags Details
List of packages used in rhevh-6.3-20121012.0 (33.18 KB, text/plain)
2013-12-03 10:09 EST, Fabian Deutsch
no flags Details

  None (edit)
Comment 1 Anand Nande 2013-11-20 08:22:09 EST
I would like to rephrase the "Description of Problem" again:
Comment 4 Yaniv Bronhaim 2013-11-20 10:30:49 EST
please attach vdsm.log and supervdsm.log after the upgrade
Comment 5 Pratik Pravin Bandarkar 2013-11-21 12:17:35 EST
I have asked custo(In reply to Yaniv Bronhaim from comment #4)
> please attach vdsm.log and supervdsm.log after the upgrade

I have asked customer for supervdsm.log file as it was not captured in Logcollector. From available logs:

---->o--->o---
2013-11-20 05:55:04,205 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-72) [6c590a72] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 4e9a4c17-3306-4fff-a2cf-260
823c135c0 Type: VDS
2013-11-20 05:55:04,215 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [6c590a72] START, SetVdsStatusVDSCommand(HostName = lome.brd-klee.dk, HostId = 4e9a4c17-3306-4fff-a2cf-260823c135c0, status
=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6173dbb2



Thread-15917162::INFO::2013-11-20 04:55:40,438::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30)
Thread-15917162::DEBUG::2013-11-20 04:55:40,439::task::495::TaskManager.Task::(_debug) Task 75a79a93-a4b5-48cc-8b88-e9841d1af799: moving from state init -> state preparing
Thread-15917162::DEBUG::2013-11-20 04:55:40,442::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`83437234-979f-4217-932a-a16c18370f3d`::Request was made in '/usr/share
/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo'
Thread-15917162::DEBUG::2013-11-20 04:55:40,443::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared'
Thread-15917162::DEBUG::2013-11-20 04:55:40,446::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user)
Thread-15917162::DEBUG::2013-11-20 04:55:40,447::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`83437234-979f-4217-932a-a16c18370f3d`::Granted request
Thread-15917162::DEBUG::2013-11-20 04:55:40,449::task::495::TaskManager.Task::(_debug) Task 75a79a93-a4b5-48cc-8b88-e9841d1af799: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared)
Thread-15917162::DEBUG::2013-11-20 04:55:40,450::task::495::TaskManager.Task::(_debug) Task 75a79a93-a4b5-48cc-8b88-e9841d1af799: ref 1 aborting False
Thread-15917162::WARNING::2013-11-20 04:55:40,463::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo
  File "/usr/share/vdsm/storage/sd.py", line 663, in isISO
  File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam
  File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__
  File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__
KeyError: 'CLASS'
Thread-15917162::DEBUG::2013-11-20 04:55:40,469::task::495::TaskManager.Task::(_debug) Task 75a79a93-a4b5-48cc-8b88-e9841d1af799: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', '
version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'iso
prefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}}


[pbandark@dhcp223-129 tunis-2013112005001384923603]$ cat sos_commands/vdsm/su_vdsm_-s_.usr.bin.python_.usr.share.vdsm.dumpStorageTable.pyc
Traceback (most recent call last):
  File "/usr/share/vdsm/dumpStorageTable.py", line 278, in <module>
  File "/usr/share/vdsm/dumpStorageTable.py", line 13, in __init__
  File "/usr/share/vdsm/dumpStorageTable.py", line 153, in _buildStorageTable
  File "/usr/share/vdsm/dumpStorageTable.py", line 56, in _getSds
KeyError: 'imageslist'


engine=#   select id,name,compatibility_version from storage_pool;
                  id                  |  name   | compatibility_version 
--------------------------------------+---------+-----------------------
 c4e5889a-348d-11e2-83fe-9f72369eab30 | Default | 3.0   (1 row)
--o<----o<---

Might be some metadata corruption issue? To confirm I have also asked to provide ISO & export metadata files.
Comment 6 Pratik Pravin Bandarkar 2013-11-21 12:20:50 EST
Created attachment 827386 [details]
vdsm.log
Comment 7 Nikhil Gupta 2013-11-26 03:41:07 EST
Created attachment 829107 [details]
iso-metadata
Comment 8 Nikhil Gupta 2013-11-26 03:41:49 EST
Created attachment 829108 [details]
export-metadata
Comment 9 Nikhil Gupta 2013-11-26 03:44:10 EST
Customer do not find supervdsm.log file in /var/log/vdsm/ directory. I have attached iso-metadata and export-metadata file provided by the customer.
Comment 10 Anand Nande 2013-11-27 01:20:34 EST
Yaniv, Please check the data provided until now and let me know if we need anything else. Thanks
Comment 11 Allon Mureinik 2013-11-27 05:36:06 EST
Fede, is this related to the pool upgrade? Seems like we changed the format there?

Thread-15917162::WARNING::2013-11-20 04:55:40,463::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo
  File "/usr/share/vdsm/storage/sd.py", line 663, in isISO
  File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam
  File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__
  File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__
KeyError: 'CLASS'
Comment 12 Federico Simoncelli 2013-11-27 11:46:37 EST
(In reply to Allon Mureinik from comment #11)
> Fede, is this related to the pool upgrade? Seems like we changed the format
> there?
> 
> Thread-15917162::WARNING::2013-11-20
> 04:55:40,463::sp::1065::Storage.StoragePool::(getInfo) Could not get full
> domain information, it is probably unavailable
> Traceback (most recent call last):
>   File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo
>   File "/usr/share/vdsm/storage/sd.py", line 663, in isISO
>   File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam
>   File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__
>   File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__
> KeyError: 'CLASS'

The domain metadata is not changed during the pool upgrade.
Looking at the two metadata attached I can see the CLASS entry. Is there any other domain in the pool?
Comment 13 Sergey Gotliv 2013-11-27 16:57:54 EST
Customer has 4 storage domains:

45cdff8b-89c0-4926-a8c7-80354064bfa6 - ISO which metadata is attached to this case. We can see that it contains the CLASS entry.

5d7b5178-3790-472c-a13b-b481eb27a61e - Export, its metadata is attached and it contains the CLASS entry either.

What about these?

6b892067-ac7e-4039-a7be-5b7387525b16

772472cb-2429-466d-96cd-00b3301ef78a

Can you, please, check?
Comment 14 Pratik Pravin Bandarkar 2013-11-28 06:12:00 EST
(In reply to Sergey Gotliv from comment #13)
> Customer has 4 storage domains:
> 
> 45cdff8b-89c0-4926-a8c7-80354064bfa6 - ISO which metadata is attached to
> this case. We can see that it contains the CLASS entry.
> 
> 5d7b5178-3790-472c-a13b-b481eb27a61e - Export, its metadata is attached and
> it contains the CLASS entry either.
> 
> What about these?
> 
> 6b892067-ac7e-4039-a7be-5b7387525b16
> 
> 772472cb-2429-466d-96cd-00b3301ef78a
> 
> Can you, please, check?


6b892067-ac7e-4039-a7be-5b7387525b16
772472cb-2429-466d-96cd-00b3301ef78a

Storage type for above domains is FC.

I think you are looking for storage domain metadata. If I am correct then, here is the file:
---
https://10.65.231.4/sysreports/1641939/00985286/log-collector-data/192.168.100.133/tunis-2013112005001384923603/sos_commands/vdsm/lvm_pvs_-v_-o_all
---
Comment 18 Anand Nande 2013-11-29 03:41:24 EST
--sorry about the flag change, setting it back--
Comment 20 Anand Nande 2013-11-29 07:10:10 EST
ok - let me check with them.
Comment 24 Fabian Deutsch 2013-12-03 10:09:06 EST
Created attachment 832144 [details]
List of packages used in rhevh-6.3-20121012.0

vdsm related versions are:

vdsm-4.9-113.4.el6_3.x86_64 (RSA/8, Fri Oct  5 15:11:43 2012, Key ID 199e2f91fd431d51)
vdsm-cli-4.9-113.4.el6_3.x86_64 (RSA/8, Fri Oct  5 15:11:45 2012, Key ID 199e2f91fd431d51)
vdsm-hook-vhostmd-4.9-113.4.el6_3.x86_64 (RSA/8, Fri Oct  5 15:11:46 2012, Key ID 199e2f91fd431d51)
vdsm-reg-4.9-113.4.el6_3.x86_64 (RSA/8, Fri Oct  5 15:11:47 2012, Key ID 199e2f91fd431d51)
Comment 34 Sergey Gotliv 2013-12-11 10:16:26 EST

*** This bug has been marked as a duplicate of bug 1031634 ***

Note You need to log in before you can comment on or make changes to this bug.