(I'm not sure if the component or the team is correct. If not, please redirect it). I'm trying to run oVirt in nested virtualization with AMD's various Zen/Zen+ based CPU's (Ryzen, Threadripper,EPYC). In Nested Virtualization mode, when trying to create or launch a VM, it stops and complains that the "monitor" flag is missing. Checking libvirt domcapabilities shows that indeed the monitor policy is "disabled" which is correct (checking against other virtualization solutions), but oVirt doesn't respect the domcapabilities. Could someone please disable the monitor flag check? it cannot be enabled and it's not a bug in the CPU or KVM.
*** Bug 1689361 has been marked as a duplicate of this bug. ***
Please attach logs (engine.log, libvirt logs, qemu logs). We don't directly check for flags outside of setting a CPU model. Is this coming from qemu? Is vdsm-hook-nestedvt in use?
Created attachment 1544682 [details] engine log
Created attachment 1544683 [details] VDSM log
I don't see any libvirt or qemu logs. Where are they? I'm enclosing both vdsm and engine logs which shows the error. Exact message is: 6464: error : virCPUx86Compare:1731 : the CPU is incompatible with host CPU: Host CPU does not provide required features: monitor Output of virsh domcapabilities: # virsh domcapabilities | grep mon <feature policy='disable' name='monitor'/>
Forgot to mention: yes, installed vdsm-hook-nestedvt and checked that that it appears in host hooks (it does).
So, that message comes directly from libvirt. Libvirt and qemu logs will be on the host the VM was scheduled on before it failed (likely to be the same host as the vdsm logs). Both are under /var/log/ Is vdsm-hook-nestedvt installed?
yes, vdsm-hook-nestedvt installed and running. /var/log/libvirt/qemu doesn't help much - it has the VM log which only shows: cat test-client.log 2019-03-16 00:59:38.003+0000: shutting down, reason=failed 2019-03-16 01:15:28.349+0000: shutting down, reason=failed 2019-03-16 01:28:57.159+0000: shutting down, reason=failed 2019-03-16 01:29:04.283+0000: shutting down, reason=failed 2019-03-16 01:29:51.729+0000: shutting down, reason=failed 2019-03-16 01:31:44.493+0000: shutting down, reason=failed /var/log/qemu-ga is an empty directory. tailing the journald when starting a VM shows: מרץ 16 03:52:44 localhost.localdomain vdsm[10633]: WARN Attempting to add an existing net user: ovirtmgmt/a3b4d8de-f2d3-4272-843c-fba78751f481 מרץ 16 03:52:45 localhost.localdomain libvirtd[6416]: 2019-03-16 01:52:45.401+0000: 6466: error : virCPUx86Compare:1731 : the CPU is incompatible with host CPU: Host CPU does not provide required features: monitor מרץ 16 03:52:45 localhost.localdomain vdsm[10633]: WARN File: /var/lib/libvirt/qemu/channels/a3b4d8de-f2d3-4272-843c-fba78751f481.ovirt-guest-agent.0 already removed מרץ 16 03:52:45 localhost.localdomain vdsm[10633]: WARN Attempting to remove a non existing network: ovirtmgmt/a3b4d8de-f2d3-4272-843c-fba78751f481 מרץ 16 03:52:45 localhost.localdomain vdsm[10633]: WARN Attempting to remove a non existing net user: ovirtmgmt/a3b4d8de-f2d3-4272-843c-fba78751f481 מרץ 16 03:52:45 localhost.localdomain vdsm[10633]: WARN File: /var/lib/libvirt/qemu/channels/a3b4d8de-f2d3-4272-843c-fba78751f481.org.qemu.guest_agent.0 already removed
Please attach /proc/cpuinfo from L0 host, and domcapabilities output. Then the same from your nested L1 host, plus its domain xml from libvirt. If you manage to start a nested guest manually, can you please also get qemu cmdline and cpuinfo from the L2 guest?
As requested, I'm including a dump of l0 and l1 cpuinfo and dom capabilities. I also include the ovirt-node1 dumpxml as well as centos7 dumpxml. I found something very interesting: On the host (Fedora 29 with Ryzen 7) I created a CentOS 7 nested guest and installed CentOS 7 below it (so: Fedora host -> Centos nested -> Centos guest without nest) - this works perfectly ok. However - I launched the ovirt node 1 (latest - 4.3.1) as a guest with nested virtualization and I tried to launch a VM using virsh (Centos 7 guest, no nested) - it stops with the CPU error about monitor. So, it seems that the problem related to the Node-NG-appliance which I installed as ovirt-node-1. On a standard CentOS geust with nested, everything works, no errors... So, how can I find what causes it in the Node-NG?
Created attachment 1544756 [details] Dump XML as requested
Just to make myself clear - all VM's were created on the host (Fedora 29) using virt-manager
After researching further, I found the following issue: I installed CentOS as L1 guest with nested virtualization, and added oVirt Repo, and started the hosted-engine deployment. It creates the HE VM, launches it and it works well (I can access it by port 6900). However, when it comes to the storage part, after giving it the NFS share and continuing deployment, it creates the new HE VM in the NFS, moving the data and then when it tries to launch the new VM - it goes up and down. So while it does this up & down, I mounted manually my virtual machines and tried to launch a VM using virsh (using virsh create) And .. surprise surprise: # virsh create nfs-server.xml Please enter your authentication name: hetz Please enter your password: error: Failed to create domain from nfs-server.xml error: the CPU is incompatible with host CPU: Host CPU does not provide required features: monitor Prior to deploying the HE on this VM, KVM inside the guest OS worked perfectly well with virsh. After the failed deployed - I got the above. I'm enclosing the whole VDSM stuff as the ansible logs doesn't show anything relevant..
Created attachment 1544820 [details] VDSM logs after running hosted-engine deployment
Update #3: When running HE as a stand alone VM (not deploying using the hosted-engine --deploy) and adding a nested VM as "node" - it creates the same issue on this new "node". Hope this helps...
Thanks, Hetz. I'll look at the logs tomorrow. vdsm does try to do CPU detection and set a host model appropriately (including HE setups -- you would have been prompted for this as part of the deployment), but we may be missing something here...
Confirmed, and I know for sure that this doesn't happen with nested Intel CPUs, since I use them regularly
Hello, any progress on this bug? I'm experiencing the same problem deploying a nested HostedEngine. If you need more logs or tests, just tell me.
(In reply to Hetz Ben Hamo from comment #0) > (I'm not sure if the component or the team is correct. If not, please > redirect it). > > I'm trying to run oVirt in nested virtualization with AMD's various Zen/Zen+ > based CPU's (Ryzen, Threadripper,EPYC). > > In Nested Virtualization mode, when trying to create or launch a VM, it > stops and complains that the "monitor" flag is missing. > > Checking libvirt domcapabilities shows that indeed the monitor policy is > "disabled" which is correct (checking against other virtualization > solutions), but oVirt doesn't respect the domcapabilities. > > Could someone please disable the monitor flag check? it cannot be enabled > and it's not a bug in the CPU or KVM. Disabling the monitor flag in the CPU capabilities works just fine on my Ryzen 3900x. I have spun up several nested VM's perfectly. Just FFI, this is the file i edited /usr/share/libvirt/cpu_map/x86_EPYC.xml and removed the "monitor" feature line and restarted libvirtd on the kvm hosts (systemd restart libvirtd). If the CPU for the cluster is set to EPYC then everything works. Let me know if anyone needs any more info.
Milan, do you happen to know why the attached patch was abandoned?
No idea, I've never seen that patch.
(In reply to Milan Zamazal from comment #21) > No idea, I've never seen that patch. Ack, thanks. Worth checking on the latest version to see if it works now.
(In reply to Arik from comment #22) > Ack, thanks. > > Worth checking on the latest version to see if it works now. targeting 4.4.1 accordingly, not leaving around a bug ON_QA without a target milestone.
The problem still occurs, Verification steps: 1. Install ovirt-engine-4.4.1.7-0.3 (HE) on AMD EPYC 7451 2. Create two RHEL8.2 VMs in the env. created in section 1 3. Install ovirt-engine-4.4.1.7-0.3 on the 1st VM (from section 2) - the nested env. 4. Add the 2nd VM as Node to the nested env. 5. Create and start a VM in the nested env. Result: - Starting VM in the nested env. fails with: 2020-07-08 14:45:05,294+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-31) [1d1885cc] EVENT_ID: VM_DOWN_ERROR(119), VM vm1_test is down with error. Exit message: the CPU is incompatible with host CPU: Host CPU does not provide required features: monitor. Additional information: - The monitor flag doesn't appear (/proc/cpuinfo) on the VMs created in section 2 - engine.log attached.
Created attachment 1700334 [details] engine.log
FYI, still occurs in 4.4.3/AV 8.3.1 as of yesterday.
virsh domcapabilities reports: <cpu> <mode name='host-passthrough' supported='yes'/> <mode name='host-model' supported='yes'> <model fallback='forbid'>EPYC-IBPB</model> <vendor>AMD</vendor> ... <feature policy='disable' name='monitor'/> ... </mode> <mode name='custom' supported='yes'> ... <model usable='yes'>Opteron_G3</model> ... </mode> </cpu> We start the VM with: <cpu match="exact"> <model>Opteron_G3</model> <topology cores="1" sockets="16" threads="1" /> <numa> <cell cpus="0-15" id="0" memory="1048576" /> </numa> </cpu> And it fails with: libvirt.libvirtError: the CPU is incompatible with host CPU: Host CPU does not provide required features: monitor When I explicitly disable `monitor' feature by adding <feature name="monitor" policy="disable"/> to <cpu> definition above, the VM starts. I don't know how the features are supposed to work and whether we should be responsible to add the given <feature> manually or whether libvirt should do the right thing itself. I.e. if it should be fixed in oVirt or in libvirt. If nobody here knows, I'll ask libvirt devs.
I already asked them.. As you can see in this BZ (https://bugzilla.redhat.com/show_bug.cgi?id=1798004#c16) the solution seems to be to simply remove the "monitor" flag from all AMD Ryzen based CPU's (Zen, ZEN+, ZEN-2, ZEN-3, Ryzen, Threadripper, EPYC) and I asked the libvirt guys few years about it and even emailed AMD engineers about it, but since I'm not from RH, it was ignored. So, if you could please ask them (the libvirt guys) to completely remove this flag, then it will help, not only in oVirt, but also in Nested-VM in Virt-Manager, Boxes etc..
Thank you for the BZ pointer, I'll try to move the things forward.
(In reply to Milan Zamazal from comment #27) > When I explicitly disable `monitor' feature by adding > > <feature name="monitor" policy="disable"/> > > to <cpu> definition above, the VM starts. This is a good workaround until libvirt is fixed. In other words, libvirt is supposed to do the right thing by itself with a host-model CPU, but unfortunately the fix on libvirt side is not as easy as it could appear so using this workaround may be a good idea. BTW, this workaround will be save to be used even if libvirt is eventually fixed.
Milan, Care to share the entire <CPU> part so I can see where to add the line that you suggested please? Thanks
(In reply to Hetz Ben Hamo from comment #28) > As you can see in this BZ > (https://bugzilla.redhat.com/show_bug.cgi?id=1798004#c16) the solution seems > to be to simply remove the "monitor" flag from all AMD Ryzen based CPU's > (Zen, ZEN+, ZEN-2, ZEN-3, Ryzen, Threadripper, EPYC) Unfortunately the fix is not that simple. We cannot just remove the "monitor" feature from all CPU models because it would break migration from new libvirt with such fix to an older version of libvirt. And such migration is explicitly required by oVirt to support upgrading individual hosts in a cluster. BTW, the new EPYC-Rome was fixed before it was released, but the existing EPYC, EPYC-IBPB, and Opteron_G3 cannot be simply fixed this way. > and I asked the libvirt guys few years about it and even emailed AMD > engineers about it, but since I'm not from RH, it was ignored. This is certainly not the way things work, I'm pretty sure you were not ignored because you are not from Red Hat.
(In reply to Jiri Denemark from comment #30) > (In reply to Milan Zamazal from comment #27) > > When I explicitly disable `monitor' feature by adding > > > > <feature name="monitor" policy="disable"/> > > > > to <cpu> definition above, the VM starts. > > This is a good workaround until libvirt is fixed. In other words, libvirt is > supposed to do the right thing by itself with a host-model CPU, but > unfortunately the fix on libvirt side is not as easy as it could appear so > using this workaround may be a good idea. BTW, this workaround will be save > to be used even if libvirt is eventually fixed. OK, thank you for the information, I think it would be good enough until libvirt is fixed. Arik, do we want to add such a workaround? Maybe providing a hook for the purpose or just documenting how to create it?
(In reply to Hetz Ben Hamo from comment #31) > Care to share the entire <CPU> part so I can see where to add the line that > you suggested please? No special trick, just: <cpu match="exact"> <model>Opteron_G3</model> <topology cores="1" sockets="16" threads="1" /> <numa> <cell cpus="0-15" id="0" memory="1048576" /> </numa> <feature name="monitor" policy="disable"/> </cpu> I think <feature> element can be added using before_vm_start Vdsm hook.
(In reply to Milan Zamazal from comment #33) > OK, thank you for the information, I think it would be good enough until > libvirt is fixed. +1 > > Arik, do we want to add such a workaround? Maybe providing a hook for the > purpose or just documenting how to create it? I think we should make it work out of the box rather than documenting it But I wonder if it wouldn't be simpler to change it on the engine side instead of introducing a hook - do we need anything that is not available on the engine side?
(In reply to Arik from comment #35) > I think we should make it work out of the box rather than documenting it OK. > But I wonder if it wouldn't be simpler to change it on the engine side > instead of introducing a hook - > do we need anything that is not available on the engine side? My idea was, since it affects only nested virtualization in certain environments, to give the users chance to add the hook in case they experience the problem and the hook can be easily removed once libvirt is fixed. But it would be indeed simpler for the users if it was handled on the Engine side transparently. The only question is whether we can define clearly under which circumstances to add the flag.
Yeah, I don't see an indication in the engine of whether or not a host is set with nested-virtualization enabled.. Would a fix on the VDSM side be just like https://gerrit.ovirt.org/#/c/98728 but to set the 'policy' to 'disable'?
(In reply to Arik from comment #37) > Yeah, I don't see an indication in the engine of whether or not a host is > set with nested-virtualization enabled.. > Would a fix on the VDSM side be just like https://gerrit.ovirt.org/#/c/98728 > but to set the 'policy' to 'disable'? Maybe, if it doesn't break other AMD CPUs. There is also `cpuflags' hook that can be used to set that flag, but unlike the patch above or an Engine solution, it requires manual configuration.
Correction: nestedvt hook is used on the bare metal host, not the VM host. So disabling the feature there doesn't help.
Verified with: - ovirt-engine-4.4.4-0.1.el8ev.noarch - vdsm-4.40.36-1.el8ev.x86_64 - libvirt-daemon-6.6.0-7.module+el8.3.0+8424+5ea525c5.x86_64 Verification steps: 1. Install ovirt-engine-4.4.4-0.1 (HE) on AMD EPYC 7451 2. Create two RHEL-8.3 VMs in the env. created in section 1 3. Install ovirt-engine-4.4.4-0.1 on the 1st VM (from section 2) - the nested env. 4. Add the 2nd VM as a host to the nested env. 5. Create and start a VM in the nested env. Result: - VM started successfully - dumpxml contains the monitor fix: <cpu mode='custom' match='exact' check='full'> <model fallback='forbid'>EPYC</model> <topology sockets='16' dies='1' cores='1' threads='1'/> <feature policy='disable' name='monitor'/> <feature policy='require' name='x2apic'/> <feature policy='require' name='hypervisor'/> <feature policy='disable' name='svm'/> <feature policy='require' name='topoext'/> <numa> <cell id='0' cpus='0-15' memory='1048576' unit='KiB'/> </numa> </cpu>
This bugzilla is included in oVirt 4.4.4 release, published on December 21st 2020. Since the problem described in this bug report should be resolved in oVirt 4.4.4 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.