Bug 1518703
| Summary: | VM fails to start with Hook error | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [oVirt] ovirt-engine | Reporter: | Lilach Zitnitski <lzitnits> | ||||
| Component: | BLL.Virt | Assignee: | Michal Skrivanek <michal.skrivanek> | ||||
| Status: | CLOSED NOTABUG | QA Contact: | meital avital <mavital> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 4.2.0 | CC: | bugs, fromani, lzitnits, ratamir | ||||
| Target Milestone: | --- | Keywords: | Regression | ||||
| Target Release: | --- | Flags: | ratamir:
blocker?
|
||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2017-11-29 14:19:04 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | Virt | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Created attachment 1360317 [details]
logs
(In reply to Lilach Zitnitski from comment #0) > Description of problem: > After updating the engine and hosts to the latest 4.2 build, VM fails to > start on any host in the cluster. > Engine log is showing the following error: > 2017-11-29 15:38:22,978+02 ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (ForkJoinPool-1-worker-12) [] EVENT_ID: VM_DOWN_ERROR(119), VM vm1 is down > with error. Exit message: Hook Error: ('',). > > Version-Release number of selected component (if applicable): > ovirt-engine-4.2.0-0.0.master.20171127212333.git11e14b9.el7.centos.noarch > vdsm-4.20.8-53.gitc3edfc0.el7.centos.x86_64 > > How reproducible: > 100% > > Steps to Reproduce: > 1. start vm > > Actual results: > VM fails to start > > Expected results: > VM should start. > > Additional info: > > engine.log > > 2017-11-29 15:38:22,978+02 ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (ForkJoinPool-1-worker-12) [] EVENT_ID: VM_DOWN_ERROR(119), VM vm1 is down wi > th error. Exit message: Hook Error: ('',). > 2017-11-29 15:38:22,980+02 INFO > [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] > (ForkJoinPool-1-worker-12) [] add VM > '70d011ec-0674-4912-8948-382feb27a861'(vm1) to rerun tr > eatment > 2017-11-29 15:38:22,984+02 ERROR > [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] > (ForkJoinPool-1-worker-12) [] Rerun VM > '70d011ec-0674-4912-8948-382feb27a861'. Called fro > m VDS 'host_mixed_2' > 2017-11-29 15:38:23,027+02 WARN > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engine-Thread-229) [] EVENT_ID: USER_INITIATED_RUN_V > M_FAILED(151), Failed to run VM vm1 on Host host_mixed_2. > 2017-11-29 15:38:23,031+02 ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engine-Thread-229) [] EVENT_ID: > USER_FAILED_RUN_VM(54), Failed to run VM vm1 (User: admin@internal-authz). > > vdsm.log > > 2017-11-29 15:37:52,420+0200 INFO (vm/70d011ec) [root] > /usr/libexec/vdsm/hooks/before_vm_start/50_vfio_mdev: rc=1 err=Traceback > (most recent call last): > File "/usr/libexec/vdsm/hooks/before_vm_start/50_vfio_mdev", line 48, in > <module> > from vdsm import supervdsm > ImportError: cannot import name supervdsm > (hooks:110) > > 2017-11-29 15:37:53,009+0200 ERROR (vm/70d011ec) [virt.vm] > (vmId='70d011ec-0674-4912-8948-382feb27a861') The vm start process failed > (vm:904) > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 833, in > _startUnderlyingVm > self._run() > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2652, in _run > self._custom) > File "/usr/lib/python2.7/site-packages/vdsm/hooks.py", line 153, in > before_vm_start > return _runHooksDir(domxml, 'before_vm_start', vmconf=vmconf) > File "/usr/lib/python2.7/site-packages/vdsm/hooks.py", line 120, in > _runHooksDir > raise exception.HookError(err) > HookError: Hook Error: ('',) Please check if you are running the same version of vdsm core (vdsm-python package) and of the vfio_mdev hook. If they indeed match, it is a bug, likely caused by ffe8b53a05dd70cd2b35aafc18d8848def69772d No, it's not the same version: vdsm-hook-vfio-mdev-4.20.6-62.gitd3023e4.el7.centos.noarch vdsm-python-4.20.8-53.gitc3edfc0.el7.centos.noarch how is 4.20.6-62 and 4.20.8-53 same versions? (In reply to Lilach Zitnitski from comment #3) > No, it's not the same version: > > vdsm-hook-vfio-mdev-4.20.6-62.gitd3023e4.el7.centos.noarch > vdsm-python-4.20.8-53.gitc3edfc0.el7.centos.noarch OK, please make sure to align the two version. We made changes in between the two versions that justify the error. If it happens again with both vdsm and hook version 4.20.8-53.gitc3edfc0.el7 , please reopen. |
Description of problem: After updating the engine and hosts to the latest 4.2 build, VM fails to start on any host in the cluster. Engine log is showing the following error: 2017-11-29 15:38:22,978+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-12) [] EVENT_ID: VM_DOWN_ERROR(119), VM vm1 is down with error. Exit message: Hook Error: ('',). Version-Release number of selected component (if applicable): ovirt-engine-4.2.0-0.0.master.20171127212333.git11e14b9.el7.centos.noarch vdsm-4.20.8-53.gitc3edfc0.el7.centos.x86_64 How reproducible: 100% Steps to Reproduce: 1. start vm Actual results: VM fails to start Expected results: VM should start. Additional info: engine.log 2017-11-29 15:38:22,978+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-12) [] EVENT_ID: VM_DOWN_ERROR(119), VM vm1 is down wi th error. Exit message: Hook Error: ('',). 2017-11-29 15:38:22,980+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-12) [] add VM '70d011ec-0674-4912-8948-382feb27a861'(vm1) to rerun tr eatment 2017-11-29 15:38:22,984+02 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (ForkJoinPool-1-worker-12) [] Rerun VM '70d011ec-0674-4912-8948-382feb27a861'. Called fro m VDS 'host_mixed_2' 2017-11-29 15:38:23,027+02 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-229) [] EVENT_ID: USER_INITIATED_RUN_V M_FAILED(151), Failed to run VM vm1 on Host host_mixed_2. 2017-11-29 15:38:23,031+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-229) [] EVENT_ID: USER_FAILED_RUN_VM(54), Failed to run VM vm1 (User: admin@internal-authz). vdsm.log 2017-11-29 15:37:52,420+0200 INFO (vm/70d011ec) [root] /usr/libexec/vdsm/hooks/before_vm_start/50_vfio_mdev: rc=1 err=Traceback (most recent call last): File "/usr/libexec/vdsm/hooks/before_vm_start/50_vfio_mdev", line 48, in <module> from vdsm import supervdsm ImportError: cannot import name supervdsm (hooks:110) 2017-11-29 15:37:53,009+0200 ERROR (vm/70d011ec) [virt.vm] (vmId='70d011ec-0674-4912-8948-382feb27a861') The vm start process failed (vm:904) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 833, in _startUnderlyingVm self._run() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2652, in _run self._custom) File "/usr/lib/python2.7/site-packages/vdsm/hooks.py", line 153, in before_vm_start return _runHooksDir(domxml, 'before_vm_start', vmconf=vmconf) File "/usr/lib/python2.7/site-packages/vdsm/hooks.py", line 120, in _runHooksDir raise exception.HookError(err) HookError: Hook Error: ('',)