Created attachment 1360317 [details]
logs

(In reply to Lilach Zitnitski from comment #0)
> Description of problem:
> After updating the engine and hosts to the latest 4.2 build, VM fails to
> start on any host in the cluster.
> Engine log is showing the following error:
> 2017-11-29 15:38:22,978+02 ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (ForkJoinPool-1-worker-12) [] EVENT_ID: VM_DOWN_ERROR(119), VM vm1 is down
> with error. Exit message: Hook Error: ('',).
> 
> Version-Release number of selected component (if applicable):
> ovirt-engine-4.2.0-0.0.master.20171127212333.git11e14b9.el7.centos.noarch
> vdsm-4.20.8-53.gitc3edfc0.el7.centos.x86_64
> 
> How reproducible:
> 100%
> 
> Steps to Reproduce:
> 1. start vm
> 
> Actual results:
> VM fails to start
> 
> Expected results:
> VM should start.
> 
> Additional info:
> 
> engine.log
> 
> 2017-11-29 15:38:22,978+02 ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (ForkJoinPool-1-worker-12) [] EVENT_ID: VM_DOWN_ERROR(119), VM vm1 is down wi
> th error. Exit message: Hook Error: ('',).
> 2017-11-29 15:38:22,980+02 INFO 
> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
> (ForkJoinPool-1-worker-12) [] add VM
> '70d011ec-0674-4912-8948-382feb27a861'(vm1) to rerun tr
> eatment
> 2017-11-29 15:38:22,984+02 ERROR
> [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring]
> (ForkJoinPool-1-worker-12) [] Rerun VM
> '70d011ec-0674-4912-8948-382feb27a861'. Called fro
> m VDS 'host_mixed_2'
> 2017-11-29 15:38:23,027+02 WARN 
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (EE-ManagedThreadFactory-engine-Thread-229) [] EVENT_ID: USER_INITIATED_RUN_V
> M_FAILED(151), Failed to run VM vm1 on Host host_mixed_2.
> 2017-11-29 15:38:23,031+02 ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (EE-ManagedThreadFactory-engine-Thread-229) [] EVENT_ID:
> USER_FAILED_RUN_VM(54), Failed to run VM vm1 (User: admin@internal-authz).
> 
> vdsm.log
> 
> 2017-11-29 15:37:52,420+0200 INFO  (vm/70d011ec) [root]
> /usr/libexec/vdsm/hooks/before_vm_start/50_vfio_mdev: rc=1 err=Traceback
> (most recent call last):
>   File "/usr/libexec/vdsm/hooks/before_vm_start/50_vfio_mdev", line 48, in
> <module>
>     from vdsm import supervdsm
> ImportError: cannot import name supervdsm
>  (hooks:110)
> 
> 2017-11-29 15:37:53,009+0200 ERROR (vm/70d011ec) [virt.vm]
> (vmId='70d011ec-0674-4912-8948-382feb27a861') The vm start process failed
> (vm:904)
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 833, in
> _startUnderlyingVm
>     self._run()
>   File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2652, in _run
>     self._custom)
>   File "/usr/lib/python2.7/site-packages/vdsm/hooks.py", line 153, in
> before_vm_start
>     return _runHooksDir(domxml, 'before_vm_start', vmconf=vmconf)
>   File "/usr/lib/python2.7/site-packages/vdsm/hooks.py", line 120, in
> _runHooksDir
>     raise exception.HookError(err)
> HookError: Hook Error: ('',)

Please check if you are running the same version of vdsm core (vdsm-python package) and of the vfio_mdev hook. If they indeed match, it is a bug, likely caused by ffe8b53a05dd70cd2b35aafc18d8848def69772d

No, it's not the same version:

vdsm-hook-vfio-mdev-4.20.6-62.gitd3023e4.el7.centos.noarch
vdsm-python-4.20.8-53.gitc3edfc0.el7.centos.noarch

how is 4.20.6-62 and 4.20.8-53 same versions?

(In reply to Lilach Zitnitski from comment #3)
> No, it's not the same version:
> 
> vdsm-hook-vfio-mdev-4.20.6-62.gitd3023e4.el7.centos.noarch
> vdsm-python-4.20.8-53.gitc3edfc0.el7.centos.noarch

OK, please make sure to align the two version. We made changes in between the two versions that justify the error. If it happens again with both vdsm and hook version 4.20.8-53.gitc3edfc0.el7 , please reopen.