Description of problem:
Any operation with virtual machines on storage domain not possible, errors like
---
2014-08-21 07:39:27,065 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (org.ovirt.thread.pool-4-thread-41) [7d28e43f] HostName = rhev-node11
2014-08-21 07:39:27,065 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (org.ovirt.thread.pool-4-thread-41) [7d28e43f] Command HotPlugDiskVDSCommand(HostName = rhev-node11, HostId = b82aa
a5b-9de2-4f42-bbcd-18c975db31ca, vmId=89e1ce3a-cf3a-4ccc-9cd2-6c505a19d4c6, diskId = e240d058-976b-4141-9fa6-57d3fbef841f) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException:
Failed to HotPlugDiskVDS, error = cannot open file '/rhev/data-center/a8436df1-3f28-4ab1-a071-ff78650fbaef/b64d40e6-71c0-4ef2-ae65-092013738dc4/images/e240d058-976b-4141-9fa6-57d3fbef841f/88f6af09-9587-45b3-9a98-62b6e578496b': No such file or directory, code = 45
2014-08-21 07:39:27,065 INFO [org.ovirt.engine.core.vd
And again
10.2.3.161-sosreport-rhev-node11-20140821061623-37f1/rhev-node11-2014082106141408601647/var/log/vdsm/vdsm.log:Thread-60909::INFO::2014-08-21 05:42:30,546::clientIF::322::vds::(prepareVolumePath) prepared volume path: /rhev/data-center/a8436df1-3f28-4ab1-a071-ff78650fbaef/b64d40e6-71c0-4ef2-ae65-092013738dc4/images/e240d058-976b-4141-9fa6-57d3fbef841f/88f6af09-9587-45b3-9a98-62b6e578496b
10.2.3.161-sosreport-rhev-node11-20140821061623-37f1/rhev-node11-2014082106141408601647/var/log/vdsm/vdsm.log:Thread-60909::DEBUG::2014-08-21 05:42:30,561::vm::1583::vm.Vm::(blockDev) vmId=`89e1ce3a-cf3a-4ccc-9cd2-6c505a19d4c6`::Unable to determine if the path '/rhev/data-center/a8436df1-3f28-4ab1-a071-ff78650fbaef/b64d40e6-71c0-4ef2-ae65-092013738dc4/images/e240d058-976b-4141-9fa6-57d3fbef841f/88f6af09-9587-45b3-9a98-62b6e578496b' is a block device
10.2.3.161-sosreport-rhev-node11-20140821061623-37f1/rhev-node11-2014082106141408601647/var/log/vdsm/vdsm.log:OSError: [Errno 2] No such file or directory: '/rhev/data-center/a8436df1-3f28-4ab1-a071-ff78650fbaef/b64d40e6-71c0-4ef2-ae65-092013738dc4/images/e240d058-976b-4141-9fa6-57d3fbef841f/88f6af09-9587-45b3-9a98-62b6e578496b'
10.2.3.161-sosreport-rhev-node11-20140821061623-37f1/rhev-node11-2014082106141408601647/var/log/vdsm/vdsm.log:Thread-60909::DEBUG::2014-08-21 05:42:30,561::vm::1583::vm.Vm::(blockDev) vmId=`89e1ce3a-cf3a-4ccc-9cd2-6c505a19d4c6`::Unable to determine if the path '/rhev/data-center/a8436df1-3f28-4ab1-a071-ff78650fbaef/b64d40e6-71c0-4ef2-ae65-092013738dc4/images/e240d058-976b-4141-9fa6-57d3fbef841f/88f6af09-9587-45b3-9a98-62b6e578496b' is a block device
---
Correct symlink exists only on one host:
[cash@dhcp-26-166 log-collector-data]$ grep -r 092013738dc4 * | grep tree
10.2.3.135/rhev-node1-2014082513191408972767/sos_commands/vdsm/su_vdsm_-s_.bin.sh_-c_.usr.bin.tree_-l_.rhev.data-center: | `-- b64d40e6-71c0-4ef2-ae65-092013738dc4
10.2.3.136/rhev-node2-2014082513191408972769/sos_commands/vdsm/su_vdsm_-s_.bin.sh_-c_.usr.bin.tree_-l_.rhev.data-center: | `-- b64d40e6-71c0-4ef2-ae65-092013738dc4
10.2.3.137/rhev-node3-2014082513191408972767/sos_commands/vdsm/su_vdsm_-s_.bin.sh_-c_.usr.bin.tree_-l_.rhev.data-center: | `-- b64d40e6-71c0-4ef2-ae65-092013738dc4
10.2.3.141/rhev-node4-2014082513191408972767/sos_commands/vdsm/su_vdsm_-s_.bin.sh_-c_.usr.bin.tree_-l_.rhev.data-center: | `-- b64d40e6-71c0-4ef2-ae65-092013738dc4
10.2.3.142/rhev-node5-2014082513211408972874/sos_commands/vdsm/su_vdsm_-s_.bin.sh_-c_.usr.bin.tree_-l_.rhev.data-center:| |-- b64d40e6-71c0-4ef2-ae65-092013738dc4 -> /rhev/data-center/mnt/glusterSD/bigdata-1.netz.xxx.de:_rvtest/b64d40e6-71c0-4ef2-ae65-092013738dc4
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Version-Release number of selected component (if applicable):
vdsm-4.14.13-2.el6ev.x86_64
rhevm-backend-3.4.2-0.2.el6ev.noarch
How reproducible:
Always for a customer. Tried with freshly reinstalled hypervisor (RHEV-H (20140821.1.el6ev)), issue persists
Steps to Reproduce:
1. Provision new hypervisor
2. Start VM
Actual results:
VM test is down. Exit message: cannot open file '/rhev/data-center/c6900106-6959-11e2-9396-00304832304c/c3d8030b-2f0d-4460-b087-ef2f154b371c/images/7961cc6f-6513-4dd3-8c8b-6d544f67988a/b96ab690-efe9-4600-a759-9dbfc9c67e87': No such file or directory.
Relevant /rhev/data-center structure
/rhev/data-center/mnt/glusterSD/bigdata-1.netz.xxx.de:_rvtest2/c3d8030b-2f0d-4460-b087-ef2f154b371c/images/7961cc6f-6513-4dd3-8c8b-6d544f67988a:
total 1025
-rw-rw----. 1 vdsm kvm 10737418240 Aug 28 07:33 b96ab690-efe9-4600-a759-9dbfc9c67e87
-rw-rw----. 1 vdsm kvm 1048576 Aug 28 07:33 b96ab690-efe9-4600-a759-9dbfc9c67e87.lease
-rw-r--r--. 1 vdsm kvm 268 Aug 28 07:33 b96ab690-efe9-4600-a759-9dbfc9c67e87.meta
Expected results:
VM starts, /rhev-datacenter structure like on a working host:
/rhev/data-center/mnt/glusterSD/bigdata-1.netz.xxx.de:_rvtest2/c3d8030b-2f0d-4460-b087-ef2f154b371c/images:
total 4
drwxr-xr-x. 2 vdsm kvm 4096 Aug 28 07:33 7961cc6f-6513-4dd3-8c8b-6d544f67988a
/rhev/data-center/mnt/glusterSD/bigdata-1.netz.xxx.de:_rvtest2/c3d8030b-2f0d-4460-b087-ef2f154b371c/images/7961cc6f-6513-4dd3-8c8b-6d544f67988a:
total 1025
-rw-rw----. 1 vdsm kvm 10737418240 Aug 28 07:33 b96ab690-efe9-4600-a759-9dbfc9c67e87
-rw-rw----. 1 vdsm kvm 1048576 Aug 28 07:33 b96ab690-efe9-4600-a759-9dbfc9c67e87.lease
-rw-r--r--. 1 vdsm kvm 268 Aug 28 07:33 b96ab690-efe9-4600-a759-9dbfc9c67e87.meta
Additional info:
workaround is to manually create the missing symlink, but VDSM recreates them after restart. permanent solution required as soon as possible