Created attachment 1398216 [details]
screenshot and full logs

Description of problem: 

My Data Center Domain storage pool has 2 Master Domain(named Stored2 on Node2, named Node1Container on Node1). and my old master domain (named DATANd01 on Node1 ) also hung on preparing for maintenance. When I tried to activate my old master domain (named DATANd01 on Node1 ) all  storage domain goes down (deactivated) and up (activating) Master Domain keep on rotating on other Data Domain.




Version-Release number of selected component (if applicable):

Ovirt Version (oVirt Engine Version: 4.1.9.1.el7.centos)


How reproducible:



Steps to Reproduce:
1. Put two hosts on maintenance then restart host
2. first Host not auto mount the shared storage domain when up.
3. Second Host mount shared storage normally.

Actual results:

Master Storage Domain goes duplicate on different storage domain in Data Center. and keep on deactivating and activating all storage domain and rotating the master domain when reconstructing master domain on Data Center 

Expected results:

Make master domain storage only 1 and stop on deactivating and activating other data domain storage

Additional info:

attached screenshot and full logs vdsm and engine

Ovirt Version (oVirt Engine Version: 4.1.9.1.el7.centos)


Event Error:
Sync Error on Master Domain between Host Node2 and oVirt Engine. Domain Stored2 is marked as master in ovirt engine Database but not on storage side Please consult with support

VDSM Node2 command ConnectStoragePoolVDS failed: Wrong Master domain or it's version: u'SD=f3e372e3-1251-4195-a4b9- 1027e40059df, pool=5a865884-0366-0330-02b8- 0000

VDSM Node2 command HSMGetAllTastsStatusesVDS failed: Not SPM: ()

Failed to deactivate Storage Domain DATANd01 (Data Center UnsecuredEnv)

Here's logs from engine:

------------------------------ ------------------------------ ------------------------------ -------------
[root@dev2engine ~]# tail /var/log/messages
Feb 20 07:01:01 dev2engine systemd: Starting Session 20 of user root.
Feb 20 07:01:01 dev2engine systemd: Removed slice User Slice of root.
Feb 20 07:01:01 dev2engine systemd: Stopping User Slice of root.
Feb 20 07:58:52 dev2engine systemd: Created slice User Slice of root.
Feb 20 07:58:52 dev2engine systemd: Starting User Slice of root.
Feb 20 07:58:52 dev2engine systemd-logind: New session 21 of user root.
Feb 20 07:58:52 dev2engine systemd: Started Session 21 of user root.
Feb 20 07:58:52 dev2engine systemd: Starting Session 21 of user root.
Feb 20 08:01:01 dev2engine systemd: Started Session 22 of user root.
Feb 20 08:01:01 dev2engine systemd: Starting Session 22 of user root.


------------------------------ ------------------------------ ------------------------------ -------------
[root@dev2engine ~]# tail /var/log/ovirt-engine/engine. log
2018-02-20 08:01:16,062+08 INFO  [org.ovirt.engine.core.bll. eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7- thread-32) [102e9d3c] Finished reconstruct for pool '5a865884-0366-0330-02b8- 0000000002d4'. Clearing event queue
2018-02-20 08:01:27,825+08 WARN  [org.ovirt.engine.core. vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-7- thread-23) [] Master domain is not in sync between DB and VDSM. Domain Stored2 marked as master in DB and not in the storage
2018-02-20 08:01:27,862+08 WARN  [org.ovirt.engine.core.bll. storage.pool. ReconstructMasterDomainCommand ] (org.ovirt.thread.pool-7- thread-23) [213f42b9] Validation of action 'ReconstructMasterDomain' failed for user SYSTEM. Reasons: VAR__ACTION__RECONSTRUCT_ MASTER,VAR__TYPE__STORAGE__ DOMAIN,ACTION_TYPE_FAILED_ STORAGE_DOMAIN_STATUS_ ILLEGAL2,$status PreparingForMaintenance
2018-02-20 08:01:27,882+08 INFO  [org.ovirt.engine.core.bll. eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7- thread-20) [929330e] Finished reconstruct for pool '5a865884-0366-0330-02b8- 0000000002d4'. Clearing event queue
2018-02-20 08:01:40,106+08 WARN  [org.ovirt.engine.core. vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-7- thread-17) [] Master domain is not in sync between DB and VDSM. Domain Stored2 marked as master in DB and not in the storage
2018-02-20 08:01:40,197+08 WARN  [org.ovirt.engine.core.bll. storage.pool. ReconstructMasterDomainCommand ] (org.ovirt.thread.pool-7- thread-17) [7af552c1] Validation of action 'ReconstructMasterDomain' failed for user SYSTEM. Reasons: VAR__ACTION__RECONSTRUCT_ MASTER,VAR__TYPE__STORAGE__ DOMAIN,ACTION_TYPE_FAILED_ STORAGE_DOMAIN_STATUS_ ILLEGAL2,$status PreparingForMaintenance
2018-02-20 08:01:40,246+08 INFO  [org.ovirt.engine.core.bll. eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7- thread-22) [73673040] Finished reconstruct for pool '5a865884-0366-0330-02b8- 0000000002d4'. Clearing event queue
2018-02-20 08:01:51,809+08 WARN  [org.ovirt.engine.core. vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-7- thread-26) [] Master domain is not in sync between DB and VDSM. Domain Stored2 marked as master in DB and not in the storage
2018-02-20 08:01:51,846+08 WARN  [org.ovirt.engine.core.bll. storage.pool. ReconstructMasterDomainCommand ] (org.ovirt.thread.pool-7- thread-26) [20307cbe] Validation of action 'ReconstructMasterDomain' failed for user SYSTEM. Reasons: VAR__ACTION__RECONSTRUCT_ MASTER,VAR__TYPE__STORAGE__ DOMAIN,ACTION_TYPE_FAILED_ STORAGE_DOMAIN_STATUS_ ILLEGAL2,$status PreparingForMaintenance
2018-02-20 08:01:51,866+08 INFO  [org.ovirt.engine.core.bll. eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7- thread-49) [2c11a866] Finished reconstruct for pool '5a865884-0366-0330-02b8- 0000000002d4'. Clearing event queue