Description of problem:

In the vdsm-4.17.23 , many storage related activities are failing with error "Image is not a legal chain" because of the incorrect handling of files under /rhev/data-center/

In a RHEV environment with more than 1 host, created a VM having below disk

engine=# select image_guid,image_group_id,parentid from images where image_group_id in   
(select image_group_id from vm_images_view where disk_id in   
(select device_id from vm_device where vm_id = (select vm_guid from vm_static where vm_name = 'test-vm') and device = 'disk'))   
order by image_group_id,parentid;
              image_guid              |            image_group_id            |               parentid               
--------------------------------------+--------------------------------------+--------------------------------------
 b93d9217-f7d4-4f3f-8b0e-14a267c63b70 | 6a5efd25-ebb6-404d-bfc3-c02f1383ff4a | 00000000-0000-0000-0000-000000000000


After creating the disk, a directory was created in the  host A which is the SPM host.

==
root@HOST-A~]# ls -l /rhev/data-center/00000001-0001-0001-0001-0000000003bd/d9b512c6-fe14-4b69-9757-ec707fdd1231/images/ |grep 6a5efd25-ebb6-404d-bfc3-c02f1383ff4a
drwxr-xr-x. 2 vdsm kvm 60 Mar 21 18:53 6a5efd25-ebb6-404d-bfc3-c02f1383ff4a
==


Started the VM in the host B which creates a symlink in HOST B.

==
root@HOST-B~]#ls -l /rhev/data-center/00000001-0001-0001-0001-0000000003bd/d9b512c6-fe14-4b69-9757-ec707fdd1231/images/ |grep 6a5efd25-ebb6-404d-bfc3-c02f1383ff4a
lrwxrwxrwx. 1 vdsm kvm 95 Mar 22 00:29 6a5efd25-ebb6-404d-bfc3-c02f1383ff4a -> /var/run/vdsm/storage/d9b512c6-fe14-4b69-9757-ec707fdd1231/6a5efd25-ebb6-404d-bfc3-c02f1383ff4a
==

Set HOST-B as SPM host manually.

Shutdown the VM.

Try to take template of this VM. It fails with below error.

===
jsonrpc.Executor/1::DEBUG::2016-03-22 00:41:21,997::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'Volume.copy' in bridge with {u'preallocate': 1, u'volFormat': 5, u'volType': 6, u'dstImgUUID': u'5b6a3cd3-6b7b-4446-815c-9c2c9a6c85da', u'dstVolUUID': u'9015e879-224c-40f0-8b22-1d2a452b1951', u'dstSdUUID': u'd9b512c6-fe14-4b69-9757-ec707fdd1231', u'volumeID': u'b93d9217-f7d4-4f3f-8b0e-14a267c63b70', u'imageID': u'6a5efd25-ebb6-404d-bfc3-c02f1383ff4a', u'postZero': u'false', u'storagepoolID': u'00000001-0001-0001-0001-0000000003bd', u'storagedomainID': u'd9b512c6-fe14-4b69-9757-ec707fdd1231', u'force': u'false', u'desc': u'{"DiskAlias":"test-vm_Disk1","DiskDescription":""}'}
jsonrpc.Executor/1::INFO::2016-03-22 00:41:22,099::image::298::Storage.Image::(isLegal) image 6a5efd25-ebb6-404d-bfc3-c02f1383ff4a in domain d9b512c6-fe14-4b69-9757-ec707fdd1231 has vollist ['b93d9217-f7d4-4f3f-8b0e-14a267c63b70']
jsonrpc.Executor/1::ERROR::2016-03-22 00:41:22,099::blockVolume::459::Storage.Volume::(validateImagePath) Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/blockVolume.py", line 457, in validateImagePath
    os.mkdir(imageDir, 0o755)
OSError: [Errno 17] File exists: '/rhev/data-center/00000001-0001-0001-0001-0000000003bd/d9b512c6-fe14-4b69-9757-ec707fdd1231/images/6a5efd25-ebb6-404d-bfc3-c02f1383ff4a'
jsonrpc.Executor/1::ERROR::2016-03-22 00:41:22,100::task::866::Storage.TaskManager.Task::(_setError) Task=`93d1adac-2970-4d50-9b1e-d4481cab9ae6`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 49, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 1815, in copyImage
    pool.validateVolumeChain(sdUUID, srcImgUUID)
  File "/usr/share/vdsm/storage/securable.py", line 77, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/share/vdsm/storage/sp.py", line 2016, in validateVolumeChain
    image.Image(self.poolPath).validateVolumeChain(sdUUID, imgUUID)
  File "/usr/share/vdsm/storage/image.py", line 734, in validateVolumeChain
    raise se.ImageIsNotLegalChain(imgUUID)
ImageIsNotLegalChain: Image is not a legal chain: (u'6a5efd25-ebb6-404d-bfc3-c02f1383ff4a',)
===

Extending the disk also fails with same error.

===
cdaf11c6-6c6a-4012-b149-44da3c290b1f::DEBUG::2016-03-22 00:34:01,441::task::905::Storage.TaskManager.Task::(_runJobs) Task=`cdaf11c6-6c6a-4012-b149-44da3c290b1f`::Task.run: running job 0: extendVolumeSize: <bound method StoragePool.extendVolumeSize of <storage.sp.StoragePool object at 0x7fce800ae1d0>> (args: (u'd9b512c6-fe14-4b69-9757-ec707fdd1231', u'6a5efd25-ebb6-404d-bfc3-c02f1383ff4a', u'b93d9217-f7d4-4f3f-8b0e-14a267c63b70', 8388608) kwargs: {})
cdaf11c6-6c6a-4012-b149-44da3c290b1f::ERROR::2016-03-22 00:34:01,524::blockVolume::459::Storage.Volume::(validateImagePath) Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/blockVolume.py", line 457, in validateImagePath
    os.mkdir(imageDir, 0o755)
OSError: [Errno 17] File exists: '/rhev/data-center/00000001-0001-0001-0001-0000000003bd/d9b512c6-fe14-4b69-9757-ec707fdd1231/images/6a5efd25-ebb6-404d-bfc3-c02f1383ff4a'
===

Also other storage activity like exporting the VM , live storage migration is failing.
  
Version-Release number of selected component (if applicable):

Red Hat Enterprise Virtualization 3.6
vdsm-4.17.23-0.el7ev.noarch
Red Hat Enterprise Virtualization Hypervisor release 7.2 (20160302.1.el7ev)

How reproducible:
100%

Steps to Reproduce:

1. Create a VM and start this VM in any host other than the SPM host.
2. Set the host where we start the VM as SPM host.
3. Shutdown the VM and try to do any storage activity like creating template , extending disk, exporting the VM 

Actual results:

The storage activity on the VM fails with "Image is not a legal chain" error.

Expected results:

The storage activity on the VM should work.

Additional info:

From the code it seems that it checks for the path and if it's not a directory, the same will be created.

==
if not os.path.isdir(imageDir):
  try:
    os.mkdir(imageDir, 0o755)
  except Exception:
    self.log.error("Unexpected error", exc_info=True)
                raise se.ImagePathError(imageDir)
==

This will be a symlink to a directory if the VM is running on the host which will work good. However in this case, an broken symlink is left behind after we shutdown the VM.

file /rhev/data-center/00000001-0001-0001-0001-0000000003bd/d9b512c6-fe14-4b69-9757-ec707fdd1231/images/6a5efd25-ebb6-404d-bfc3-c02f1383ff4a
/rhev/data-center/00000001-0001-0001-0001-0000000003bd/d9b512c6-fe14-4b69-9757-ec707fdd1231/images/6a5efd25-ebb6-404d-bfc3-c02f1383ff4a: broken symbolic link to `/var/run/vdsm/storage/d9b512c6-fe14-4b69-9757-ec707fdd1231/6a5efd25-ebb6-404d-bfc3-c02f1383ff4a'

file /var/run/vdsm/storage/d9b512c6-fe14-4b69-9757-ec707fdd1231/6a5efd25-ebb6-404d-bfc3-c02f1383ff4a
/var/run/vdsm/storage/d9b512c6-fe14-4b69-9757-ec707fdd1231/6a5efd25-ebb6-404d-bfc3-c02f1383ff4a: cannot open (No such file or directory)

If this host becomes SPM and any storage related activity is done for this  VM, it fails because the creation of directory fails.