Bug 1395530

We'll need both engine and VDSM logs to look into this issue.

(In reply to Yaniv Kaul from comment #1)
> We'll need both engine and VDSM logs to look into this issue.

hello Yaniv,here is the engine log message,I now have no VDSM log.



2016-11-08 15:36:47,550 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] (pool-7-thread-2) [7723568f] Failed in 'GetVolumeInfoVDS' method
2016-11-08 15:36:47,551 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] (pool-7-thread-2) [7723568f] Command 'GetVolumeInfoVDSCommand(HostName = fedora138, GetVolumeInfoVDSCommandParameters:{expectedEngineErrors='[VolumeDoesNotExist]', runAsync='true', hostId='25f5f5cc-2791-4fe0-8983-0950a1cf8aba', storagePoolId='581bd8ed-0388-00ab-0230-0000000003df', storageDomainId='c8c44eb2-d43c-4653-bf7f-9f14c65e20be', imageGroupId='87861656-4f4d-4e1c-aafd-2504b8b58a40', imageId='c7b3cb09-0aac-4906-93ae-4c9183318411'})' execution failed: VDSGenericException: VDSErrorException: Failed to GetVolumeInfoVDS, error = Volume does not exist: (u'c7b3cb09-0aac-4906-93ae-4c9183318411',), code = 201
2016-11-08 16:35:48,479 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.TimeBoundPollVDSCommand] (org.ovirt.thread.pool-8-thread-7) [3d15bacb] Error: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: Connection issues during send request
2016-11-08 16:35:48,480 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.TimeBoundPollVDSCommand] (org.ovirt.thread.pool-8-thread-7) [3d15bacb] Exception: java.util.concurrent.ExecutionException: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: Connection issues during send request
2016-11-08 16:35:48,500 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.TimeBoundPollVDSCommand] (org.ovirt.thread.pool-8-thread-7) [3d15bacb] Command 'TimeBoundPollVDSCommand(HostName = fedora156, TimeBoundPollVDSCommandParameters:{runAsync='true', hostId='40d5aca3-2811-44b9-a94c-88c136fe3f88'})' execution failed: java.util.concurrent.ExecutionException: org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcRunTimeException: Connection issues during send request
2016-11-08 16:35:58,991 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-8-thread-7) [1b237522] Command 'PollVDSCommand(HostName = fedora156, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='40d5aca3-2811-44b9-a94c-88c136fe3f88'})' execution failed: VDSGenericException: VDSNetworkException: Timeout during xml-rpc call
2016-11-08 16:35:58,991 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-8-thread-7) [1b237522] Timeout waiting for VDSM response: Internal timeout occured
2016-11-08 16:36:01,494 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-8-thread-7) [1b237522] Command 'PollVDSCommand(HostName = fedora156, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='40d5aca3-2811-44b9-a94c-88c136fe3f88'})' execution failed: VDSGenericException: VDSNetworkException: Timeout during xml-rpc call
2016-11-08 16:36:01,494 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-8-thread-7) [1b237522] Timeout waiting for VDSM response: Internal timeout occured
2016-11-08 16:36:03,767 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "93ccb7b4-2eb8-4cf0-ac8d-0caa4121a5ec"
2016-11-08 16:36:03,997 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-8-thread-7) [1b237522] Command 'PollVDSCommand(HostName = fedora156, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='40d5aca3-2811-44b9-a94c-88c136fe3f88'})' execution failed: VDSGenericException: VDSNetworkException: Timeout during xml-rpc call
2016-11-08 16:36:03,997 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-8-thread-7) [1b237522] Timeout waiting for VDSM response: Internal timeout occured
2016-11-08 16:36:04,500 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "104d8c7e-ade2-4544-b161-0b6c37abfdb6"
2016-11-08 16:36:04,519 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "b4a1e720-567a-44c1-bd35-86e01c9c1f7d"
2016-11-08 16:39:09,751 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [41ae1020] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM fedora156 command failed: Message timeout which can be caused by communication issues
2016-11-08 16:39:09,752 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [41ae1020] Command 'ConnectStorageServerVDSCommand(HostName = fedora156, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='40d5aca3-2811-44b9-a94c-88c136fe3f88', storagePoolId='581bd8ed-0388-00ab-0230-0000000003df', storageType='NFS', connectionList='[StorageServerConnections:{id='7dee829c-cf4c-4a95-aad0-1ea5ded28e85', connection='192.168.71.138:/var/nfsshare', iqn='null', vfsType='null', mountOptions='null', nfsVersion='V3', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='40ac547d-6fc7-497d-835d-be4ca0ccf762', connection='192.168.71.138:/var/nfsshare2', iqn='null', vfsType='null', mountOptions='null', nfsVersion='V3', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='b2d71c5e-dc84-48be-bc1d-3c12a9575374', connection='192.168.71.138:/var/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='V3', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'})' execution failed: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues
2016-11-08 16:39:09,753 ERROR [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [41ae1020] Command 'org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues (Failed with error VDS_NETWORK_ERROR and code 5022)
2016-11-08 16:39:12,326 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-30) [] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM fedora156 command failed: Cannot find master domain: u'spUUID=581bd8ed-0388-00ab-0230-0000000003df, msdUUID=067eecb8-df39-48d5-97ae-cebd6ea570e4'
2016-11-08 16:39:12,327 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-30) [] Command 'ConnectStoragePoolVDSCommand(HostName = fedora156, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='40d5aca3-2811-44b9-a94c-88c136fe3f88', vdsId='40d5aca3-2811-44b9-a94c-88c136fe3f88', storagePoolId='581bd8ed-0388-00ab-0230-0000000003df', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=581bd8ed-0388-00ab-0230-0000000003df, msdUUID=067eecb8-df39-48d5-97ae-cebd6ea570e4'
2016-11-08 16:39:12,327 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-8-thread-30) [] Could not connect host 'fedora156' to pool 'Default': null



The host1 is "fedora138",host2 is "fedora156" and cluster use default.

I first add a storage domain in host "fedora138".

thanks

(In reply to hsiuming from comment #0)
> Description of problem:
> 
> I now have a ovirt cluster which has one engine and two nodes hardware.The
> nodes were installed by fedora os system,but I now have a issue when add
> host into the cluster.
> If I add these two node into the cluster as hosts,it was successful and
> working fine,then I set the storage domain in the cluster there were have a
> issue about uuid occur.(The storage type of storage domain is NFS)
> 
> The issue message is "VDSM $host2 command failed: can not find master domain
> :u'spUUID=581bd8ed-0388-00ab-0230-0000000003df,msdUUID=067eecb8-df39-48d5-
> 97ae-cebd6ea570e4".
> 
> I set the storage domain on $host1,and the $host2 will have this issue.

Hi hsiuming,
can you please describe the scenario step by step? what do you mean by "set the storage domain"?

I'd appreciate if you could attach the engine.log file from the relevant time (the log you've added is too short).

thanks,
Liron

Created attachment 1222957 [details]
enginelog

(In reply to Liron Aravot from comment #3)
> (In reply to hsiuming from comment #0)
> > Description of problem:
> > 
> > I now have a ovirt cluster which has one engine and two nodes hardware.The
> > nodes were installed by fedora os system,but I now have a issue when add
> > host into the cluster.
> > If I add these two node into the cluster as hosts,it was successful and
> > working fine,then I set the storage domain in the cluster there were have a
> > issue about uuid occur.(The storage type of storage domain is NFS)
> > 
> > The issue message is "VDSM $host2 command failed: can not find master domain
> > :u'spUUID=581bd8ed-0388-00ab-0230-0000000003df,msdUUID=067eecb8-df39-48d5-
> > 97ae-cebd6ea570e4".
> > 
> > I set the storage domain on $host1,and the $host2 will have this issue.
> 
> Hi hsiuming,
> can you please describe the scenario step by step? what do you mean by "set
> the storage domain"?
> 
> I'd appreciate if you could attach the engine.log file from the relevant
> time (the log you've added is too short).
> 
> thanks,
> Liron

Hello Liron,I already upload the log file.

We have three machine ,one installed as engine and two as hosts.

In engine web,I first set these two host machine as host.

In this step it's OK,but when I select one host NFS server path as storage domain,there were error about uuid in another host.

(In reply to hsiuming from comment #5)
> (In reply to Liron Aravot from comment #3)
> > (In reply to hsiuming from comment #0)
> > > Description of problem:
> > > 
> > > I now have a ovirt cluster which has one engine and two nodes hardware.The
> > > nodes were installed by fedora os system,but I now have a issue when add
> > > host into the cluster.
> > > If I add these two node into the cluster as hosts,it was successful and
> > > working fine,then I set the storage domain in the cluster there were have a
> > > issue about uuid occur.(The storage type of storage domain is NFS)
> > > 
> > > The issue message is "VDSM $host2 command failed: can not find master domain
> > > :u'spUUID=581bd8ed-0388-00ab-0230-0000000003df,msdUUID=067eecb8-df39-48d5-
> > > 97ae-cebd6ea570e4".
> > > 
> > > I set the storage domain on $host1,and the $host2 will have this issue.
> > 
> > Hi hsiuming,
> > can you please describe the scenario step by step? what do you mean by "set
> > the storage domain"?
> > 
> > I'd appreciate if you could attach the engine.log file from the relevant
> > time (the log you've added is too short).
> > 
> > thanks,
> > Liron
> 
> Hello Liron,I already upload the log file.
> 
> We have three machine ,one installed as engine and two as hosts.
> 
> In engine web,I first set these two host machine as host.
> 
> In this step it's OK,but when I select one host NFS server path as storage
> domain,there were error about uuid in another host.

I find the problem is that I set the storage domain and host-node on the same machine.

If the machine which is set as host-node,another machine can't mount the storage domain(NFS type) on the same machine. 

If I want to set up storage domain and host-node on the same machine,are there any method?

(In reply to hsiuming from comment #6)
> (In reply to hsiuming from comment #5)
> > (In reply to Liron Aravot from comment #3)
> > > (In reply to hsiuming from comment #0)
> > > > Description of problem:
> > > > 
> > > > I now have a ovirt cluster which has one engine and two nodes hardware.The
> > > > nodes were installed by fedora os system,but I now have a issue when add
> > > > host into the cluster.
> > > > If I add these two node into the cluster as hosts,it was successful and
> > > > working fine,then I set the storage domain in the cluster there were have a
> > > > issue about uuid occur.(The storage type of storage domain is NFS)
> > > > 
> > > > The issue message is "VDSM $host2 command failed: can not find master domain
> > > > :u'spUUID=581bd8ed-0388-00ab-0230-0000000003df,msdUUID=067eecb8-df39-48d5-
> > > > 97ae-cebd6ea570e4".
> > > > 
> > > > I set the storage domain on $host1,and the $host2 will have this issue.
> > > 
> > > Hi hsiuming,
> > > can you please describe the scenario step by step? what do you mean by "set
> > > the storage domain"?
> > > 
> > > I'd appreciate if you could attach the engine.log file from the relevant
> > > time (the log you've added is too short).
> > > 
> > > thanks,
> > > Liron
> > 
> > Hello Liron,I already upload the log file.
> > 
> > We have three machine ,one installed as engine and two as hosts.
> > 
> > In engine web,I first set these two host machine as host.
> > 
> > In this step it's OK,but when I select one host NFS server path as storage
> > domain,there were error about uuid in another host.
> 
> I find the problem is that I set the storage domain and host-node on the
> same machine.
> 
> If the machine which is set as host-node,another machine can't mount the
> storage domain(NFS type) on the same machine. 
> 
> If I want to set up storage domain and host-node on the same machine,are
> there any method?

Indeed - please use Gluster for a hyper-converged solution.
I'm closing this issue for now, as this (NFS) is not a supported configuration