Description of problem: VM with disk on iscsi on PPC environment fails to start on host with the following errors: Engine.log: 2020-07-02 09:18:50,521+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-17) [14c64a7d] EVENT_ID: VM_DOWN_ERROR(119), VM vm_TestCase5138_0209130682 is down with error. Exit message: Wake up from hibernation failed:internal error: child reported (status=125): unable to set security context 'system_u:object_r:virt_content_t:s0' on '/rhev/data-center/mnt/blockSD/97370c99-3549-4686-8536-d067f82e5daf/images/1bc19190-6f96-4d7e-9db3-a25c87451aaa/f5dd8cc0-7491-4574-a066-00ffcc0382fb': No such file or directory. 2020-07-02 09:18:50,527+03 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (ForkJoinPool-1-worker-17) [14c64a7d] Rerun VM 'ad70f664-cf00-4390-a8dc-ccb267dd283f'. Called from VDS 'host_mixed_2' 2020-07-02 09:18:50,537+03 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-17496) [14c64a7d] EVENT_ID: USER_INITIATED_RUN_VM_FAILED(151), Failed to run VM vm_TestCase5138_0209130682 on Host host_mixed_2. VDSM.log: 2020-07-02 09:18:48,754+0300 ERROR (vm/ad70f664) [virt.vm] (vmId='ad70f664-cf00-4390-a8dc-ccb267dd283f') The vm start process failed (vm:871) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/virt/vm.py", line 801, in _startUnderlyingVm self._run() File "/usr/lib/python3.6/site-packages/vdsm/virt/vm.py", line 2570, in _run fname, srcDomXML, libvirt.VIR_DOMAIN_SAVE_PAUSED) File "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/common/function.py", line 94, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python3.6/site-packages/libvirt.py", line 4851, in restoreFlags if ret == -1: raise libvirtError ('virDomainRestoreFlags() failed', conn=self) libvirt.libvirtError: internal error: child reported (status=125): unable to set security context 'system_u:object_r:v irt_content_t:s0' on '/rhev/data-center/mnt/blockSD/97370c99-3549-4686-8536-d067f82e5daf/images/1bc19190-6f96-4d7e-9db 3-a25c87451aaa/f5dd8cc0-7491-4574-a066-00ffcc0382fb': No such file or directory Saw it in our automation. After it fails, it starts on another host. In single host environment, the VM fails to start in the first attempt. Version-Release number of selected component (if applicable): rhv-4.4.1-5: vdsm-4.40.20-1.el8ev.ppc64le ovirt-engine-4.4.1.5-0.17.el8ev.noarch How reproducible: 100% Steps to Reproduce: 1. Create VM with iSCSI disk 2. Run the VM 3. Create memory snapshot 4. Power off the VM 5. Preview snapshot 6. Run the VM Actual results: Operation fails Expected results: Operation should succeed. Additional info: Logs are attached.
*** Bug 1853192 has been marked as a duplicate of this bug. ***
Seems related to the fix for bug 1840609 and bug 1842894 Liran, can you please have a look?
I don't think it's related to the above bug. In those we went back to the way it was with older libvirt (4.3 rhv). Did you start seeing it in the last build only? Evelina, can you check the path /rhev/data-center/mnt/blockSD/97370c99-3549-4686-8536-d067f82e5daf/images/1bc19190-6f96-4d7e-9db 3-a25c87451aaa/f5dd8cc0-7491-4574-a066-00ffcc0382fb? Does it exist? What is the ownership on this? Please also provide libvirt debug logs.
(In reply to Liran Rotenberg from comment #3) > I don't think it's related to the above bug. In those we went back to the > way it was with older libvirt (4.3 rhv). > Did you start seeing it in the last build only? > > Evelina, can you check the path > /rhev/data-center/mnt/blockSD/97370c99-3549-4686-8536-d067f82e5daf/images/ > 1bc19190-6f96-4d7e-9db > 3-a25c87451aaa/f5dd8cc0-7491-4574-a066-00ffcc0382fb? Does it exist? What is > the ownership on this? > > Please also provide libvirt debug logs. lrwxrwxrwx. 1 vdsm kvm 78 Jul 2 09:17 f5dd8cc0-7491-4574-a066-00ffcc0382fb -> /dev/97370c99-3549-4686-8536-d067f82e5daf/f5dd8cc0-7491-4574-a066-00ffcc0382fb
Created attachment 1700189 [details] libvirt-log
Created attachment 1700191 [details] qemu-log
Please fill severity.
Does it still happen with the latest version?
(In reply to Arik from comment #9) > Does it still happen with the latest version? Yes, tried on engine-4.4.1.8-0.7.el8ev
Can you please provide engine and vdsm logs?
(In reply to Arik from comment #11) > Can you please provide engine and vdsm logs? Don't have available PPC env at the moment. I'll add relevant logs when I'll have PPC env.
Created attachment 1711036 [details] vdsm+engine logs ovirt-engine-4.4.1.10-0.1.el8ev.noarch vdsm-4.40.22-1.el8ev.ppc64le
Created attachment 1711087 [details] full_logs I managed to reproduce it on the PPC environment and take the logs including libvirt debug logs. The VM starting on host1 in the logs and failing. libvirt-client-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64 qemu-kvm-4.2.0-29.module+el8.2.1+7297+a825794d.ppc64le kernel-4.18.0-193.el8.ppc64le selinux-policy-3.14.3-41.el8.noarch After a close look, I didn't find anything related to the engine or VDSM. But, the SELinux on the hosts are set to Enforced. I tried on a normal environment(x86_64) when setting the SELinux to Enforced and it fails on the same error.
Michal, can you please take a look? Or change it to someone who is more relevant?
(In reply to Liran Rotenberg from comment #15) > Michal, can you please take a look? Or change it to someone who is more > relevant? The attached libvirtd.log is not debug log really. It was produced using log level = INFO instead of DEBUG. Can you please attach debug logs using the settings from https://wiki.libvirt.org/page/DebugLogs ? But my rough guess is that the file doesn't exist when the domain is being resumed but it exists afterwards, when checking. E.g. a NFS is mounted afterwards? I don't think that mounts are reflected into domain namespace. One way to check if my theory is true is to disable namespaces (set namespaces=[] in qemu.conf).
Ah. This looks like bug 1772838 then. Thing is, the restore path is a block device (see comment 5) and as such it is not created in the namespace. If we want this fixed in RHEL-AV-8.2* then I guess we need to ask PMs.
We are now using RHEL 8.3 with RHV 4.4.3. Moving to ON_QA.
Verified with TestCase5138 on PPC environment with rhv-4.4.3-8.
This bugzilla is included in oVirt 4.4.3 release, published on November 10th 2020. Since the problem described in this bug report should be resolved in oVirt 4.4.3 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.