Bug 1176417

Created attachment 971740 [details]
logs from host and engine

Description of problem:
I got to a scenario in which after reconstruct master succeeded and the new master became active, the host didn't take SPM.
I tried to maintenance the host and nothing happened. Tried also to manually select it as SPM, didn't help either.

Version-Release number of selected component (if applicable):
host and engine: rhel6.6

rhev 3.5 vt13.4

rhevm-3.5.0-0.26.el6ev.noarch

vdsm-4.16.8.1-4.el6ev.x86_64
libvirt-0.10.2-46.el6_6.2.x86_64
qemu-kvm-rhev-0.12.1.2-2.448.el6.x86_64
selinux-policy-3.7.19-260.el6_6.1.noarch
sanlock-2.8-1.el6.x86_64


How reproducible:
Tested once

Steps to Reproduce:
Have 2 storage domains in DC, 1 NFS (master) and 1 iSCSI. One host in DC
1. Block connectivity to master domain, wait for reconstruct to finish
2. Once reconstruct is funuished, put the host in maintenance.

Actual results:

Reconstruct finished successfully:

2014-12-21 16:14:25,781 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-1) [5cb67019] Correlation ID: 5cb67019, Job ID: d27e7a73-5517-400a-9271-164f2d00
d666, Call Stack: null, Custom Event ID: -1, Message: Reconstruct Master Domain for Data Center Default completed.

Set the host in maintenance:

2014-12-21 16:16:28,655 INFO  [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-7-thread-3) [39640578] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affe
cted :  ID: ede38ee8-6f30-42d9-afa1-274b45dcbf11 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
2014-12-21 16:16:28,658 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-3) [39640578] START, SetVdsStatusVDSCommand(HostName = green-vdsc, HostId = ede38ee8-6f30-42d9
-afa1-274b45dcbf11, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 6e7ae0c3



The maintenance host operation never completed. The host remains 'up' and there is no SPM in the DC. The master domain is active.



[root@green-vdsc images]# vdsClient -s 0 getSpmStatus 00000002-0002-0002-0002-0000000000f0
        spmId = -1
        spmStatus = Free
        spmLver = -1


[root@green-vdsc images]# vdsClient -s 0 getStorageDomainInfo 0151588f-b974-4a3d-996b-4518bfc4bcfa
        uuid = 0151588f-b974-4a3d-996b-4518bfc4bcfa
        vguuid = AjWjsa-OIJU-nwSc-1xi4-035a-mGRp-AFjBif
        state = OK
        version = 3
        role = Master
        type = ISCSI
        class = Data
        pool = ['00000002-0002-0002-0002-0000000000f0']
        name = iscsi1


[root@green-vdsc images]# vdsClient -s 0 getVGInfo AjWjsa-OIJU-nwSc-1xi4-035a-mGRp-AFjBif

============================
vgsize = 58653147136 
============================
name = 0151588f-b974-4a3d-996b-4518bfc4bcfa 
============================
vgUUID = AjWjsa-OIJU-nwSc-1xi4-035a-mGRp-AFjBif 
============================
pvlist:
vendorID = XtremIO  capacity = 58653147136  fwrev = 0000  vgUUID = AjWjsa-OIJU-nwSc-1xi4-035a-mGRp-AFjBif  pathlist = [{'connection': '10.35.146.129', 'iqn': 'iqn.2008-05.com.xtremio:001e675b8ee0', 'portal': '1', 'port': '3260', 'initiatorname': 'default'}, {'connection': '10.35.146.161', 'iqn': 'iqn.2008-05.com.xtremio:001e675b8ee1', 'portal': '1', 'port': '3260', 'initiatorname': 'default'}, {'connection': '10.35.146.193', 'iqn': 'iqn.2008-05.com.xtremio:001e675ba170', 'portal': '1', 'port': '3260', 'initiatorname': 'default'}, {'connection': '10.35.146.225', 'iqn': 'iqn.2008-05.com.xtremio:001e675ba171', 'portal': '1', 'port': '3260', 'initiatorname': 'default'}]  pathstatus = [{'physdev': 'sdd', 'type': 'iSCSI', 'state': 'active', 'lun': '5'}, {'physdev': 'sdf', 'type': 'iSCSI', 'state': 'active', 'lun': '5'}, {'physdev': 'sdh', 'type': 'iSCSI', 'state': 'active', 'lun': '5'}, {'physdev': 'sdj', 'type': 'iSCSI', 'state': 'active', 'lun': '5'}]  devtype = iSCSI  pvUUID = uvHCoM-pMgK-VIIv-yOyr-yc6V-yDAS-ISRujw  serial = SXtremIO_XtremApp_PSNT_Not_Set  GUID = 3514f0c5447600864  devcapacity = 59055800320  productID = XtremApp 
============================
state = OK 
============================
vgfree = 54492397568 
============================
type = 3 
============================
attr = {'partial': '-', 'permission': 'w', 'allocation': 'n', 'exported': '-', 'clustered': '-', 'resizeable': 'z'} 




I tried also to manually set the host as SPM, didn't work. 

Waited for around 1 hour, nothing had changed.

Expected results:
DC should never be in a state where there is an active master doamain with no SPM.

Additional info: logs from host and engine