Bug 1102530

My first bug submit, so please excuse where I have missed importatnt stuff.

Description of problem:
I have recently tried to migrate my ovirt 3.3.4 + NFS environment to ovirt 3.4.1 and Gluster.

During that process, I have had issues during import whereby the gluster storage appears to be active in the ovirt gui, but the link to the gluster storage in /rhev/data-center is missing, and the import therefore fails. 

The error always seems to boild down to the following error on engine.log (the following specifically for the create of a new disk image on a gluster storage domain):

2014-05-20 08:51:21,136 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [4637af09] Correlation ID: 2b0b55ab, Job ID: 1a583643-e28a-4f09-a39d-46e4fc6d20b8, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of rhel-7_Disk1 was initiated on VM rhel-7 by peter.harris.
2014-05-20 08:51:21,137 INFO  [org.ovirt.engine.core.bll.SPMAsyncTask] (ajp--127.0.0.1-8702-9) [4637af09] BaseAsyncTask::startPollingTask: Starting to poll task 720b4d92-1425-478c-8351-4ff827b8f728.
2014-05-20 08:51:28,077 INFO  [org.ovirt.engine.core.bll.AsyncTaskManager] (DefaultQuartzScheduler_Worker-19) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now
2014-05-20 08:51:28,084 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-19) Failed in HSMGetAllTasksStatusesVDS method
2014-05-20 08:51:28,085 INFO  [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-19) SPMAsyncTask::PollTask: Polling task 720b4d92-1425-478c-8351-4ff827b8f728 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'cleanSuccess'.
2014-05-20 08:51:28,104 ERROR [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-19) BaseAsyncTask::LogEndTaskFailure: Task 720b4d92-1425-478c-8351-4ff827b8f728 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended with failure:^M
-- Result: cleanSuccess^M
-- Message: VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error = [Errno 2] No such file or directory: '/rhev/data-center/06930787-a091-49a3-8217-1418c5a9881e/967aec77-46d5-418b-8979-d0a86389a77b/images/7726b997-7e58-45f8-a5a6-9cb9a689a45a', code = 100,^M
-- Exception: VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error = [Errno 2] No such file or directory: '/rhev/data-center/06930787-a091-49a3-8217-1418c5a9881e/967aec77-46d5-418b-8979-d0a86389a77b/images/7726b997-7e58-45f8-a5a6-9cb9a689a45a', code = 100

Certainly, if I check /rhev/data-center/06930787-a091-49a3-8217-1418c5a9881e/ on the SPM server, there is no 967aec77-46d5-418b-8979-d0a86389a77b subdirectory. The only elements I have are NFS mounts.

There appear to be no errors in the SPM vdsm.log for this disk


Last nightm I also had events whereby vms on one node paused due to unknown storage error:
2014-05-28 20:09:22,788 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-40) domain 615647e2-1f60-47e1-8e55-be9f7ead6f15:strg-vm_infrastructure in problem. vds: vmhost2
2014-05-28 20:10:38,646 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) VM inf-ipa02 9175e0c0-6ebd-4882-b6b1-0a50c265c683 moved from Up --> Paused
2014-05-28 20:10:38,706 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM inf-ipa02 has paused due to unknown storage error.
2014-05-28 20:10:38,711 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) VM inf-zenoss ceb362eb-22f4-488b-bd7b-5db4aa423411 moved from Up --> Paused
2014-05-28 20:10:38,723 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM inf-zenoss has paused due to unknown storage error.
2014-05-28 20:10:38,727 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) VM sup-CMS 3cbbbe18-6eaf-47d2-b085-0eda878642cd moved from Up --> Paused
2014-05-28 20:10:38,748 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM sup-CMS has paused due to storage I/O problem.
2014-05-28 20:10:38,810 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-46) Domain 615647e2-1f60-47e1-8e55-be9f7ead6f15:strg-vm_infrastructure recovered from problem. vds: vmhost2
2014-05-28 20:10:38,811 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-46) Domain 615647e2-1f60-47e1-8e55-be9f7ead6f15:strg-vm_infrastructure has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer.

Checking this morning, and /rhev/data-center link has disappeared over night. on most of the nodes.

Is there a way of reactivating these links without going through the process of:
- Stop All VMs
- set ALL gluster nodes to maintenance in the data center (NOTE: NFS links are fine)
- Activate All gluster nodes
- restart VMs.

This migration is dead in the water if I can't get this to work, or at least recover with out VM stop/start

Version-Release number of selected component (if applicable):
- gluster 3.5.0 Release 2.el6
- vdsm 4.14.8.1 Release 0.el6
- ovirt-engine 3.4.1 Release 1.el6


How reproducible:
I am not sure how to replicate this I am afraid. It sometimes occurs when I am doing an import, and that could possibly be when I try and import multiple VMs. Importing VMs one by one seems to fail less often. So may be a network timeout?


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

This appears to be a reoccurence of 
https://bugzilla.redhat.com/show_bug.cgi?id=1069772

My environment is as follows:
OLD
vmhost1 - ovirt 3.3.4 - NFS storages

NEW
ovirtmgr - ovirt 3.4.1 (virt only setup) - gluster storage domains
vmhost2 - cluster1
vmhost3 - cluster2
vmhost4 - cluster2
vmhost5 - cluster3
vmhost6 - cluster3

My gluster volumes are created via gluster command line and I have

All hosts are running scientific Linux 6.5, and the intention is to migrate vmhost1 to new environment cluster1.

I have an NFS export storage domain which I am using to migrate VMs from vmhost1.
Volume Name: vol-vminf
Type: Distributed-Replicate
Volume ID: b0b456bb-76e9-42e7-bb95-3415db79d631
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: vmhost3:/storage/inf/br-inf
Brick2: vmhost4:/storage/inf/br-inf
Brick3: vmhost5:/storage/inf/br-inf
Brick4: vmhost6:/storage/inf/br-inf
Options Reconfigured:
storage.owner-gid: 36
storage.owner-uid: 36
server.allow-insecure: on

Volume Name: vol-vmimages
Type: Distribute
Volume ID: 91e2cf8b-2662-4c26-b937-84b8f5b62e2b
Status: Started
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: vmhost3:/storage/vmimages/br-vmimages
Brick2: vmhost3:/storage/vmimages/br-vmimages
Brick3: vmhost3:/storage/vmimages/br-vmimages
Brick4: vmhost3:/storage/vmimages/br-vmimages
Options Reconfigured:
storage.owner-gid: 36
storage.owner-uid: 36
server.allow-insecure: on