Red Hat Bugzilla – Bug 1543103
Call vdsm 'after_vm_pause' hooks when the VM has been paused because an I/O Error
Last modified: 2018-05-15 13:55:30 EDT
Description of problem: vdsm does not call 'after_vm_pause' hooks when the VM has been paused because an I/O Error How reproducible: Always Steps to Reproduce: 1. Install a vdsm hook in "/usr/libexec/hooks/after_vm_pause" directory 2. Force a VM to be paused because an I/O Error Actual results: The hook in "/usr/libexec/hooks/after_vm_pause" directory is not executed Expected results: All hooks in "/usr/libexec/hooks/after_vm_pause" directory are executed
WARN: Bug status (ON_QA) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{'rhevm-4.2-ga': '?'}', ] For more info please contact: rhv-devops@redhat.comINFO: Bug status (ON_QA) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{'rhevm-4.2-ga': '?'}', ] For more info please contact: rhv-devops@redhat.com
Hi Miguel, I've tested it on version vdsm-4.20.18-1.el7ev.x86_64 & rhvm-4.2.2-0.1.el7.noarch (rhel7.5) and see that the problem is not resolved. could you please see my steps and say maybe I miss something in reproducing. 1. Put some python or shell script (tried both) in /usr/libexec/vdsm/hooks/after_vm_pause on Host where VM is running. See that script under HostHooks in Admin Portal (attached after_vm_pause_hook.png). 2. block the iscsi storage on the same host which causes VM I/O Error Pause (I use HA VM , no lease), then unblock which causes VM resume. Result: the script placed in /usr/libexec/vdsm/hooks/after_vm_pause is not executed.
Created attachment 1398569 [details] after_vm_pause_hook.png
can you attach vdsm.log please?
please see the vdsm & engine logs attached in vdsm_engine.tar.gz in vdsm.log please log starting from: 2018-02-25 09:19:56,634+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:573) in engine.log starting from: 2018-02-25 09:19:03,277+02 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-5) [] EVENT_ID: USER_RUN_VM(32), VM golden_env_mixed_virtio_2_0 started on Host host_mixed_1 2018-02-25 09:23:40,047+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-7) [] EVENT_ID: VM_PAUSED_EIO(145), VM golden_env_mixed_virtio_2_0 has been paused due to storage I/O problem.
Created attachment 1400533 [details] vdsm_engine.tar.gz
in the log you can see it tried to execute it 2018-02-25 09:23:39,844+0200 INFO (libvirt/events) [root] /usr/libexec/vdsm/hooks/after_vm_pause/test_hook.py: rc=2 err=/usr/libexec/vdsm/hooks/a fter_vm_pause/test_hook.py: line 1: import: command not found I guess you missed the #! to run the right interpreter
Yes, you are right. I just checked that the script is executed manually and did't add the #!/usr/bin/python2. Now it is added. Still not executed upon i/o pause/resume . the error in vdsm log: 2018-02-27 10:30:11,084+0200 DEBUG (event/26) [root] FINISH thread <Thread(event/26, started daemon 139786127886080)> (concurrent:195) 2018-02-27 10:30:11,111+0200 DEBUG (libvirt/events) [root] FAILED: <err> = 'taskset: failed to execute /usr/libexec/vdsm/hooks/after_vm_pause/test_hook.py: No such file or directory\n'; <rc> = 1 (commands:86) 2018-02-27 10:30:11,113+0200 INFO (libvirt/events) [root] /usr/libexec/vdsm/hooks/after_vm_pause/test_hook.py: rc=1 err=taskset: failed to execute /usr/libexec/vdsm/hooks/after_vm_pause/test_hook.py: No such file or directory (hooks:110)
well, it's still not executable. Either way, there is an attempt to run it so the bug is verified:)
(In reply to Michal Skrivanek from comment #13) > well, it's still not executable. Either way, there is an attempt to run it > so the bug is verified:) just to confirm: I've put some simple python script /usr/libexec/vdsm/hooks/after_vm_pause/50_create_file (chmod 755) which is manually executed ok (the script is below). The vdsm recognize the created hook (2018-02-28 09:49:46,511+0200 INFO (libvirt/events) [root] /usr/libexec/vdsm/hooks/after_vm_pause/50_create_file: rc=0 err= (hooks:110)), but the script itself is not executed. #!/usr/bin/python import os d = os.path.dirname(__file__) # directory of script p = r'{}/results/hook_test'.format(d) # path to be created try: os.makedirs(p) except OSError: pass Please approve if it is enough for this bug verification
dunno, you could have probably just copied an existing code. Maybe the file extension or imports are not correct, but that's really not the point of this verification, the hook mechanism is the same for all the hooks. I think you've verified this bug good enough
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:1489