Created attachment 1784814 [details] XML Cluster Level 4.5 Description of problem: When upgrading a Windows 2012(R2) VM from cluster level 4.5 to 4.6, it causes Windows to see the VirtIO NIC as a new NIC after a reboot. This causes Windows to default again to DHCP on that NIC, and the VM loses its connectivity. A manual intervention is needed to fix this (reconfigure the NIC) I only noticed this on Windows 2012(R2) machines. But newer might be affected also but untested! Version-Release number of selected component (if applicable): OS Version: RHEL - 8.4.2105.0 - 3.el8 OS Description: oVirt Node 4.4.6 Kernel Version: 4.18.0 - 301.1.el8.x86_64 KVM Version: 5.2.0 - 16.el8s LIBVIRT Version: libvirt-7.0.0-14.el8s VDSM Version: vdsm-4.40.60.6-1.el8 How reproducible: Always Steps to Reproduce: 1. Create a Windows 2012 R2 VM with VirtIO NIC on Cluster 4.5 2. Shutdown 3. Change cluster level to 4.6 4. Boot the VM 5. The NIC is now detected as a new NIC with default settings Shutting down the VM again, and change the cluster level to 4.5 makes the old NIc to become active again. Expected results: The NIC should not change. Surely on Windows this causes issues. Additional info: In attachment the XML of the VM in Cluster level 4.5 and 4.6. The diff is really minor.
Created attachment 1784816 [details] XML Cluster Level 4.6
Looking at the 'interface' element in the domain XML on 4.6 cluster, I don't see anything suspicious: <interface type='bridge'> <mac address='00:16:3e:7e:e5:b0'/> <source bridge='VLAN_111'/> <bandwidth> <inbound average='128000' peak='384000' burst='1024000'/> <outbound average='128000' peak='384000' burst='1024000'/> </bandwidth> <target dev='vnet31'/> <model type='virtio'/> <driver name='vhost' queues='2'/> <filterref filter='vdsm-no-mac-spoofing'/> <link state='up'/> <mtu size='1500'/> <alias name='ua-d0fae394-5963-4c47-a001-f3910377c896'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> so moving to the network team to check this further
The documentation text flag should only be set after 'doc text' field is provided. Please provide the documentation text and set the flag to '?' again.
(In reply to Jean-Louis Dupond from comment #0) > Created attachment 1784814 [details] > XML Cluster Level 4.5 > > Description of problem: > When upgrading a Windows 2012(R2) VM from cluster level 4.5 to 4.6, it > causes Windows to see the VirtIO NIC as a new NIC after a reboot. > This causes Windows to default again to DHCP on that NIC, and the VM loses > its connectivity. > A manual intervention is needed to fix this (reconfigure the NIC) > > I only noticed this on Windows 2012(R2) machines. But newer might be > affected also but untested! > > > Version-Release number of selected component (if applicable): > OS Version: > RHEL - 8.4.2105.0 - 3.el8 > OS Description: > oVirt Node 4.4.6 > Kernel Version: > 4.18.0 - 301.1.el8.x86_64 > KVM Version: > 5.2.0 - 16.el8s > LIBVIRT Version: > libvirt-7.0.0-14.el8s > VDSM Version: > vdsm-4.40.60.6-1.el8 What is exactly installed on the host where this VM run? Above I can se RHEL version, but also oVirt Node version. And I can also see libvirt/qemu from CentOS Stream Anyway could you please provide full logs from the host gathered by sos log-collector?
The only difference I was able to find was the 'target' element: <target dev='vnet17'/> vs <target dev='vnet31'/>. On Linux VM, especially with predictable names, this won't be an issue, as the interface name is mainly decided from PCI address. I am not an expert on Windows, but it seems to me that the target might be related to this.
(In reply to Martin Perina from comment #4) > > What is exactly installed on the host where this VM run? > Above I can se RHEL version, but also oVirt Node version. And I can also see > libvirt/qemu from CentOS Stream > > Anyway could you please provide full logs from the host gathered by sos > log-collector? Its ovirt-node: ovirt-node-ng-image-update-4.4.6.1-1.el8.noarch So nothing exotic :) I'll check to provide more deteails from sos-collector if needed. (In reply to Ales Musil from comment #5) > The only difference I was able to find was the 'target' element: <target > dev='vnet17'/> vs <target dev='vnet31'/>. > > On Linux VM, especially with predictable names, this won't be an issue, as > the interface name is mainly decided > from PCI address. > > I am not an expert on Windows, but it seems to me that the target might be > related to this. The target is always changing, even when migrating between hosts afaik. This because the vnet is just incrementing depending on the amount of vm's running and their amount of nics. It shouldn't make any difference to the VM itself.
(In reply to Ales Musil from comment #5) > The only difference I was able to find was the 'target' element: <target > dev='vnet17'/> vs <target dev='vnet31'/>. I believe we do not write target at all and leave it on libvirt. However that should not change during migrations. Weird. But indeed it shouldn't matter. Anyway, you should look at virtual hardware, probably. And the windows drivers. That's what's different between CLs
Created attachment 1785184 [details] Regdump Attached a small regdump which shows how the NIC's are seen inside the Windows guest. As you see both have same PCI path etc.
Digging a bit further. Changed the Cluster level to 4.6, but the machine type to pc-q35-rhel8.3.0. And the NIC keeps working. So it seems like something in the machine type change caused the NIC to be seen as a new NIC inside the guest. We can see in the guest that the ContainerID of the NIC changed, and thus Windows defines it as a new NIC. Now when checking whats different inside Qemu for pc-q35-rhel8.3.0 vs pc-q35-rhel8.4.0. It seems the diff is rather small. But the most important one (which might trigger some ACPI changes): diff --git a/hw/i386/pc.c b/hw/i386/pc.c index f3fc695fe2f..d5ea5b634c5 100644 --- a/hw/i386/pc.c +++ b/hw/i386/pc.c @@ -363,11 +363,15 @@ GlobalProperty pc_rhel_compat[] = { { TYPE_X86_CPU, "vmx-exit-load-perf-global-ctrl", "off" }, /* bz 1508330 */ { "vfio-pci", "x-no-geforce-quirks", "on" }, - /* BZ 1846886 */ - { "ICH9-LPC", "x-smi-cpu-hotplug", "off" }, }; const size_t pc_rhel_compat_len = G_N_ELEMENTS(pc_rhel_compat); Could this be the cause?
(In reply to Jean-Louis Dupond from comment #9) > Digging a bit further. > > Changed the Cluster level to 4.6, but the machine type to pc-q35-rhel8.3.0. > And the NIC keeps working. Short version: one has to keep the same machine type for existing guests to be sure nothing changes. (I think libvirt does expand/embed machine type in persistent domain XML for you), if you do not use that, than you'll have to re-implement this logic yourselves. > So it seems like something in the machine type change caused the NIC to be > seen as a new NIC inside the guest. Long version: You are probably seeing effects of https://bugzilla.redhat.com/show_bug.cgi?id=1934158 Where virtio drivers enumerate NIC differently with corrected PCI bus description in ACPI tables. It's the issue with virtio drivers (other drivers are fine with this change), I've opened https://bugzilla.redhat.com/show_bug.cgi?id=1939546 to explore possibility of the fix in drivers, you can ask Vadim if there are any plans to work on it. Meanwhile QEMU has implemented workaround to restore broken PCI bus UID on old machine types (read 8.3 and older), so existing Windows guests with static IP won't loose network configuration. That means that if one moves affected guest image to from QEMU-8.3 to 8.4, without keeping machine type version, virtio NIC will enumerate as different device and has to be (re)configured.
Thanks a lot Igor for the details. It seems indeed we are affected by this. Now from oVirt side I think we always default to the most recent Machine Type. So every windows 2012+? user upgrading its cluster level and have the default machine type will be affected by this. First I think we need to have some red flag about this in the release notes of 4.4.6 to at least warn about this issue. Second we might want to review the 'newest machine type by default' policy?
(In reply to Jean-Louis Dupond from comment #11) > Thanks a lot Igor for the details. It seems indeed we are affected by this. > > Now from oVirt side I think we always default to the most recent Machine > Type. > So every windows 2012+? user upgrading its cluster level and have the > default machine type will be affected by this. > > First I think we need to have some red flag about this in the release notes > of 4.4.6 to at least warn about this issue. > Second we might want to review the 'newest machine type by default' policy? It's fine to use newest machine type for new VMs, but as for the existing VMs it's better to stick to machine types that were used for creating them (most of the time existing image works fine when moved to newest machine type, but there aren't any guaranties that it will not break). Machine types are there to maintain guest ABI stable across different QEMU version and ensure that it's possible to migrate VM between various QEMU versions.
(In reply to Jean-Louis Dupond from comment #11) > Thanks a lot Igor for the details. It seems indeed we are affected by this. > > Now from oVirt side I think we always default to the most recent Machine > Type. we don't. we use a fixed machine type for concrete cluster level. We use different machine types in different CLs, when switching CLs it is expected that things change. You're not forced to change it, we support 4.2 onwards. > So every windows 2012+? user upgrading its cluster level and have the > default machine type will be affected by this. > > First I think we need to have some red flag about this in the release notes > of 4.4.6 to at least warn about this issue. > Second we might want to review the 'newest machine type by default' policy?
(In reply to Igor Mammedov from comment #12) > It's fine to use newest machine type for new VMs, but as for the existing VMs > it's better to stick to machine types that were used for creating them > (most of the time existing image works fine when moved to newest machine > type, > but there aren't any guaranties that it will not break). > > Machine types are there to maintain guest ABI stable across different QEMU > version and ensure that it's possible to migrate VM between various > QEMU versions. Sure, but the management is expected to update existing VMs as you upgrade your cluster level. As Michal pointed out above, the emulated machine is fixed for a given cluster-level and you need to manually ask to upgrade the cluster for the cluster-level to change. But once you do this (upgrade your cluster), we should use advance the emulated machine also of existing VMs, not just for new VMs. The rational is that by upgrading your cluster, you get all the new features and fixes. And if you have big clusters, you are not expected to reprovision them or to change them one-by-one, but let the cluster upgrade change them altogether. Sure, this changes the ABI but it shouldn't lead to such issues within the guest
typoes.. Sure, but the management is expected to update existing VMs as you upgrade your cluster level. As Michal pointed out above, the emulated machine is fixed for a given cluster-level and you need to manually ask to upgrade the cluster for the cluster-level to change. But once you do this (upgrade your cluster), we should advance the emulated machine also of existing VMs, not just for new VMs. The rational is that by upgrading your cluster, you get all the new features and fixes. And if you have big clusters in terms of number of VMs, you are not expected to reprovision those VMs or to change them one-by-one but let the cluster upgrade change them at once. Sure, this changes the ABI but it shouldn't lead to such issues within the guest..
typos :D
(In reply to Arik from comment #16) > typoes.. > > Sure, but the management is expected to update existing VMs as you upgrade > your cluster level. > As Michal pointed out above, the emulated machine is fixed for a given > cluster-level and you need to manually ask to upgrade the cluster for the > cluster-level to change. > But once you do this (upgrade your cluster), we should advance the emulated > machine also of existing VMs, not just for new VMs. > The rational is that by upgrading your cluster, you get all the new features > and fixes. after QEMU upgrade, old machines types also get fixes, modulo ones that break ABI/migration, and more often than not old machine types can use new features as well (though one should not use them if migration to older QEMU is expected later on) >And if you have big clusters in terms of number of VMs, you are > not expected to reprovision those VMs or to change them one-by-one but let > the cluster upgrade change them at once. Though for guests it mostly just works when you update machine type to the latest, there aren't any guaranties that it will work in every case as expected. (you are practically moving guest from one hw to another and expect guest to reconfigure magically). That mostly depends on guest OS being able to transparently adapt to the changes, but that's not always possible (at least not the way one would wish to). > Sure, this changes the ABI but it shouldn't lead to such issues within the > guest.. So how do you expect this to work when guest OS can't adapt to the changed hardware the way you would like it to?
(In reply to Igor Mammedov from comment #18) > (In reply to Arik from comment #16) > > typoes.. > > > > Sure, but the management is expected to update existing VMs as you upgrade > > your cluster level. > > As Michal pointed out above, the emulated machine is fixed for a given > > cluster-level and you need to manually ask to upgrade the cluster for the > > cluster-level to change. > > But once you do this (upgrade your cluster), we should advance the emulated > > machine also of existing VMs, not just for new VMs. > > The rational is that by upgrading your cluster, you get all the new features > > and fixes. > after QEMU upgrade, old machines types also get fixes, > modulo ones that break ABI/migration, and more often than not > old machine types can use new features as well > (though one should not use them if migration to older QEMU is expected later > on) Yeah and that's part of what I meant earlier, the management layer simplifies this complexity for users - we don't make this distinction between fixes that break ABI/migration or not and features that apply to old machine types or not the cluster level abstracts all that - if you use the latest cluster level you should get all the fixes and features for VMs that were started after the cluster was upgraded (even if those VMs were created before the upgrade) and should be able to migrate them to all other hosts in that cluster (cluster == migration domain) > > >And if you have big clusters in terms of number of VMs, you are > > not expected to reprovision those VMs or to change them one-by-one but let > > the cluster upgrade change them at once. > > Though for guests it mostly just works when you update machine type to the > latest, > there aren't any guaranties that it will work in every case as expected. > (you are practically moving guest from one hw to another and expect guest > to reconfigure magically). > That mostly depends on guest OS being able to transparently adapt to the > changes, > but that's not always possible (at least not the way one would wish to). > > > Sure, this changes the ABI but it shouldn't lead to such issues within the > > guest.. > > So how do you expect this to work when guest OS can't adapt to the changed > hardware the way you would like it to? Sure, when users upgrade their clusters they should be aware that their VMs may be affected by this - we change the SKU, we sometimes replace deprecated devices, we change the emulated machine, ... And it's ok that we can't guarantee the guest OS would adapt to the changes (though we do our best to prevent things that may lead to that - like changing i440fx to q35 for guest operating systems that don't support it) My point is that you previously mentioned a possible change in virtio-win drivers that could prevent this and if it's possible, we should rather do it instead of asking users/customers to try to keep the original emulated machine of their VMs - there are stories about users that still run VMs that were created more than 10 years ago, their original emulated machine is no longer even supported on RHEL 8.. :)
(In reply to Arik from comment #19) > (In reply to Igor Mammedov from comment #18) > > (In reply to Arik from comment #16) > > > typoes.. > > > > > > Sure, but the management is expected to update existing VMs as you upgrade > > > your cluster level. > > > As Michal pointed out above, the emulated machine is fixed for a given > > > cluster-level and you need to manually ask to upgrade the cluster for the > > > cluster-level to change. > > > But once you do this (upgrade your cluster), we should advance the emulated > > > machine also of existing VMs, not just for new VMs. > > > The rational is that by upgrading your cluster, you get all the new features > > > and fixes. > > after QEMU upgrade, old machines types also get fixes, > > modulo ones that break ABI/migration, and more often than not > > old machine types can use new features as well > > (though one should not use them if migration to older QEMU is expected later > > on) > > Yeah and that's part of what I meant earlier, the management layer > simplifies this complexity for users - we don't make this distinction > between fixes that break ABI/migration or not and features that apply to old > machine types or not > the cluster level abstracts all that - if you use the latest cluster level > you should get all the fixes and features for VMs that were started after > the cluster was upgraded (even if those VMs were created before the upgrade) > and should be able to migrate them to all other hosts in that cluster > (cluster == migration domain) > > > > > >And if you have big clusters in terms of number of VMs, you are > > > not expected to reprovision those VMs or to change them one-by-one but let > > > the cluster upgrade change them at once. > > > > Though for guests it mostly just works when you update machine type to the > > latest, > > there aren't any guaranties that it will work in every case as expected. > > (you are practically moving guest from one hw to another and expect guest > > to reconfigure magically). > > That mostly depends on guest OS being able to transparently adapt to the > > changes, > > but that's not always possible (at least not the way one would wish to). > > > > > Sure, this changes the ABI but it shouldn't lead to such issues within the > > > guest.. > > > > So how do you expect this to work when guest OS can't adapt to the changed > > hardware the way you would like it to? > > Sure, when users upgrade their clusters they should be aware that their VMs > may be affected by this - we change the SKU, we sometimes replace deprecated > devices, we change the emulated machine, ... > And it's ok that we can't guarantee the guest OS would adapt to the changes > (though we do our best to prevent things that may lead to that - like > changing i440fx to q35 for guest operating systems that don't support it) > > My point is that you previously mentioned a possible change in virtio-win > drivers that could prevent this and if it's possible, we should rather do it > instead of asking users/customers to try to keep the original emulated > machine of their VMs I would agree with you, unfortunately in real world scenario it just doesn't work. No one can guarantee that there won't be any bugs or unexpected by user behavior in drivers or other software they use in VM. Let's imagine virtio drivers will be 'fixed', and some users in cluster update them (which likely would require user's intervention) and some don't. Then when machine type is upgraded (unconditionally by cloud operator) to the newest type, the VMs with old drivers will be broken (ultimately the source of issue might be not drivers but something else where it's impossible to update) That's one of use-cases where using old machine types would prevent customers from experiencing issues. I'm not saying that drivers shouldn't be fixed (they should). However approach of changing hw of existing VMs and expecting guest OS being fine with it, is playing games of chance. Upgrade of machine type should be VM's owner choice, not cloud operator. PS: If I were user, My first reaction to operator unilateraly breaking previously working VM, would be to look for a hosting that doesn't take down my service and sue for damages. As for users with VMs that use phased out machine types, they should plan and test upgrade path like with any hw upgrade as any sane sysadmin would do. [...]
(In reply to Igor Mammedov from comment #20) > I would agree with you, unfortunately in real world scenario it just doesn't > work. > No one can guarantee that there won't be any bugs or unexpected by user > behavior > in drivers or other software they use in VM. > > Let's imagine virtio drivers will be 'fixed', and some users in cluster > update > them (which likely would require user's intervention) and some don't. Then > when > machine type is upgraded (unconditionally by cloud operator) to the newest > type, > the VMs with old drivers will be broken (ultimately the source of issue > might be not > drivers but something else where it's impossible to update) > > That's one of use-cases where using old machine types would prevent customers > from experiencing issues. I'm not saying that drivers shouldn't be fixed > (they should). > > However approach of changing hw of existing VMs and expecting guest OS being > fine with it, is playing games of chance. Upgrade of machine type should be > VM's > owner choice, not cloud operator. Yes, it might be different for clouds, my comments above refer only to a virtualization platform that is intended for "pet VMs" I see the risk in changing the hardware but that's what oVirt does for more than a decade and we had to do it in order to enable users to upgrade their RHEL 6 based data centers to RHEL 8 over the years. And I agree with you that it would have been better to keep the original emulated machine if we could > > PS: > If I were user, > My first reaction to operator unilateraly breaking previously working VM, > would be to look for a hosting that doesn't take down my service and sue for > damages. Right and fortunately we had no reports of completely breaking previously working VMs when upgrading the cluster level, our QE are expected to test that when they test cluster upgrade There were issues here and there though that were fixed - it would be better not to introduce them on the first place but as you wrote we can't guarantee that (because the ABI changes) so those issues should just be prioritized when identified..
What I think might be a better solution is something like the following: - When we create a VM, it should assign the current (depending on cluster level) machine type to the VM. - If cluster is upgraded, the machine type for new VM's should change to the newest one, but 'old' VM's should keep their defined machine type - We provide some scripts/ui/whatever to upgrade machine types in bulk (for example after some manual tests on particular vm's to see if everything keeps working). This way we can for example upgrade all VM's with machine type x to the latest. Or all Windows 2012+ VM's, etc - When cluster is updated oVirt should also check if there are machine types that are to old to support on new cluster level. So those can/need to be checked first. This way I think its the safest way to provide new features for new VM's, but keep compatibility for older/existing VM's. Because not upgrading the cluster level also makes you can't use the new features even on brand new VM's neither. The way its implemented now (always upgrade all vm's to the latest machine type) is just a bit risky in terms of compatibility. The machine type per vm, independent on the cluster version, is also the way how ESXi handles it for example.
(In reply to Jean-Louis Dupond from comment #22) > What I think might be a better solution is something like the following: > > - When we create a VM, it should assign the current (depending on cluster > level) machine type to the VM. > - If cluster is upgraded, the machine type for new VM's should change to the > newest one, but 'old' VM's should keep their defined machine type > - We provide some scripts/ui/whatever to upgrade machine types in bulk (for > example after some manual tests on particular vm's to see if everything > keeps working). This way we can for example upgrade all VM's with machine > type x to the latest. Or all Windows 2012+ VM's, etc > - When cluster is updated oVirt should also check if there are machine types > that are to old to support on new cluster level. So those can/need to be > checked first. > > This way I think its the safest way to provide new features for new VM's, > but keep compatibility for older/existing VM's. So why don't you set the VMs you'd like to keep running with their previous emulated machines with a custom emulated machine before upgrading the cluster? that would prevent the emulated machine from changing when the cluster is upgraded > Because not upgrading the cluster level also makes you can't use the new > features even on brand new VM's neither. You can upgrade the data center without upgrading the cluster and set new VMs with custom emulated machine = that of the new data center's compatibility level This would allow you to consume features and fixes without upgrading the cluster, but you'll need to make sure that you have upgraded hosts in the cluster that are able to run those VMs > > The way its implemented now (always upgrade all vm's to the latest machine > type) is just a bit risky in terms of compatibility. > The machine type per vm, independent on the cluster version, is also the way > how ESXi handles it for example. As I wrote above, you can achieve those things if you really want to but it requires a bit more work. As for changing this to be the default behaviour, I don't see a reason for changing that at this point. Users are used to the way it works (for more than a decade, really) and we rarely have issues with that procedure. All we need in my opinion is to avoid changes that break or significantly change existing VMs from our side
(In reply to Arik from comment #23) > > So why don't you set the VMs you'd like to keep running with their previous > emulated machines with a custom emulated machine before upgrading the > cluster? that would prevent the emulated machine from changing when the > cluster is upgraded > That's an option indeed. But I think in practice that will mean you end up setting all the VM's to a custom emulated machine, update the cluster, and then (if not forgotten, end up on some low prio todo list) set their level to the default again. Cause if you only want a custom emulated machine for VM's/OS'es that give issues, you might spend way to much time (surely in environments running multiple different OS'es etc) on testing the change before you can upgrade cluster level. > As I wrote above, you can achieve those things if you really want to but it > requires a bit more work. > As for changing this to be the default behaviour, I don't see a reason for > changing that at this point. Users are used to the way it works (for more > than a decade, really) and we rarely have issues with that procedure. All we > need in my opinion is to avoid changes that break or significantly change > existing VMs from our side Agree! Best way would indeed be that nothing breaks :) The only disadvantage in the current way is that if there is really a breaking change in Qemu with some OS'es for example, there is no clean/easy way to handle it. Or we should provide some option/script/whatever to bulk-edit affected VM's and set them to a custom emulated machine with a big warning in the changelog. But the biggest risk I think is that some breaking changes, that for example only affect some OS'es will be left unnoticed until the cluster was upgraded. Let's take for example this bug .. The change was not really mentioned in the changelog or whatever. So unknown for most of oVirt users I think. A cluster upgrade was done, and machines were not rebooted directly (which is almost always the case). Then on a good day, its Windows Update release day, suddenly during the night, all the VM's reboot and lose network access ... This is a big issue imo. There are 2 ways on handling this: 1) Don't upgrade cluster and test OS per OS with a newer machine type/cluster level (can you run a VM with cluster level 4.6 on a 4.5 cluster? Cause cluster level change often changes more then only the machine type?). New VM's need to specify the new cluster level when creating. And when all OS'es were tested and approved, the whole cluster can be upgraded. 2) Upgrade cluster does by default only change the cluster level/machine type for new vm's. Old VM's need to be tested OS by OS for example, and can then be (bulk) changed to the newest level.
(In reply to Jean-Louis Dupond from comment #24) > (In reply to Arik from comment #23) > > > > So why don't you set the VMs you'd like to keep running with their previous > > emulated machines with a custom emulated machine before upgrading the > > cluster? that would prevent the emulated machine from changing when the > > cluster is upgraded > > > > That's an option indeed. > But I think in practice that will mean you end up setting all the VM's to a > custom emulated machine, update the cluster, and then (if not forgotten, end > up on some low prio todo list) set their level to the default again. > Cause if you only want a custom emulated machine for VM's/OS'es that give > issues, you might spend way to much time (surely in environments running > multiple different OS'es etc) on testing the change before you can upgrade > cluster level. > > > > As I wrote above, you can achieve those things if you really want to but it > > requires a bit more work. > > As for changing this to be the default behaviour, I don't see a reason for > > changing that at this point. Users are used to the way it works (for more > > than a decade, really) and we rarely have issues with that procedure. Switching to the latest and greatest by default is fine for testing environment or if cluster setup is just a plaything. It's not suitable for production though, where stability matters the most. As a customer I care not about software cloud provider uses, as long as it keeps my VMs running without issues. Some customers might tolerate current behavior but some would eventually migrate to a competitor that gives them desirable stability. 'pet' VMs users even more agile, the first sign of difficult to resolve trouble and they move somewhere else. I used to run VM on google cloud, but not anymore due to unreasonable charging behavior. I'm not coming back to them any time soon and guess what recommendation it would get from me if someone asks for an advice. For enterprise users everything happens more slowly but that are users who puts stability before everything else. If we break by default their clusters and start forcing them to reconfigure all VMs to a custom ones, they will look for an alternatives. I used to be a sysadmin in my earlier carrier, if software causes problems for users or is too much of maintenance, I'd research alternatives and ask competitors for trial/competing offers of a solution that would work better. Eventually bad reputation / not met expectations accumulate in community leading to less users adopting the product. > > All we need in my opinion is to avoid changes that break or significantly change > > existing VMs from our side Practice shows that's not viable approach, We do introduce breaking changes on new machine types and we will continue to do so as it there exactly for that purpose (whether it's a new feature or an ABI breaking fix). Sometimes we even break old machine types (like with this issue, that was reported upstream half a year after the the breaking change) just in time to save RHEL8.3 and older machine types from disaster. > Agree! Best way would indeed be that nothing breaks :) > The only disadvantage in the current way is that if there is really a > breaking change in Qemu with some OS'es for example, there is no clean/easy > way to handle it. > Or we should provide some option/script/whatever to bulk-edit affected VM's > and set them to a custom emulated machine with a big warning in the > changelog. maybe the default upgrade policy should be a conservative/safe one with an ability to opt in (a separate knob) into aggressive upgrade for a selected subset of VMs for those who don't care about possible issues it could cause. (with some dialog window notifying them about consequences) > But the biggest risk I think is that some breaking changes, that for example > only affect some OS'es will be left unnoticed until the cluster was upgraded. > Let's take for example this bug .. > The change was not really mentioned in the changelog or whatever. So unknown > for most of oVirt users I think. > A cluster upgrade was done, and machines were not rebooted directly (which > is almost always the case). > Then on a good day, its Windows Update release day, suddenly during the > night, all the VM's reboot and lose network access ... This is a big issue > imo. > > There are 2 ways on handling this: > 1) Don't upgrade cluster and test OS per OS with a newer machine > type/cluster level (can you run a VM with cluster level 4.6 on a 4.5 > cluster? Cause cluster level change often changes more then only the machine > type?). > New VM's need to specify the new cluster level when creating. > And when all OS'es were tested and approved, the whole cluster can be > upgraded. It might work ror typical VMs but not for every possible config (like this case), where one doesn't know in advance what to test. Ultimately it doesn't have to be OS, it might be even a 3rd party software that user runs in VM. That's why I was saying it should be VM owner who plans and does machine type upgrade. > 2) Upgrade cluster does by default only change the cluster level/machine > type for new vm's. > Old VM's need to be tested OS by OS for example, and can then be (bulk) > changed to the newest level. Or alternatively left alone, if their machine type is still supported in new cluster or sandboxed in old cluster with owner notified about it being phased out until owner upgrades VMs to a newer machine type that can be moved to the new cluster. PS: Even if virtio drivers are fixed it won't help for existing setups unless ovirt has an ability to force drivers update. And even if there were a way for updating guests, then some new breaking change might not even have a fix to deploy. I consider this BZ as ovirt bug which unreasonably expects stability while upgrading machine type for existing guest.
(In reply to Igor Mammedov from comment #25) > I consider this BZ as ovirt bug which unreasonably expects stability while > upgrading > machine type for existing guest. And that's ok, thanks Igor for your feedback and insights on this, the only request from my side would be to prioritize fixing those issues and trying to prevent them in advance. I understand it's not always possible but from the oVirt's point of view, we can't really pin VMs to the emulated machine they were created with so we need to find the right way to keep oVirt work as good as it works for so many years with this approach.
(In reply to Arik from comment #26) > > And that's ok, thanks Igor for your feedback and insights on this, the only > request from my side would be to prioritize fixing those issues and trying > to prevent them in advance. > I understand it's not always possible but from the oVirt's point of view, we > can't really pin VMs to the emulated machine they were created with so we > need to find the right way to keep oVirt work as good as it works for so > many years with this approach. You can specify the emulated machine in the VM settings. I think now when its 'null' it uses the default (aka latest version). But it shouldn't be a big change to just put the latest version there on creation?
(In reply to Jean-Louis Dupond from comment #27) > (In reply to Arik from comment #26) > > > > And that's ok, thanks Igor for your feedback and insights on this, the only > > request from my side would be to prioritize fixing those issues and trying > > to prevent them in advance. > > I understand it's not always possible but from the oVirt's point of view, we > > can't really pin VMs to the emulated machine they were created with so we > > need to find the right way to keep oVirt work as good as it works for so > > many years with this approach. > > You can specify the emulated machine in the VM settings. > I think now when its 'null' it uses the default (aka latest version). > But it shouldn't be a big change to just put the latest version there on > creation? Technically, it's not a big change but conceptually it is. We do want our users to use the default settings in the cluster - that's what we test and that brings great value in terms of maintaining a data center over time. I see that we now question this because we have this issue in the transition from 8.3 to 8.4 and Igor clearly has a different view on it in terms of stability vs. other criteria, but at this point I really don't see a good reason to change it unless we expect bunch of regressions by keep doing this. As for your proposal in comment 24 and the feedback on it in comment 25, I agree with Igor that it may not solve everything and thus may not worth the effort. As mentioned before, even if you keep the emulated machine the same and upgrade your hosts you get new features that may cause issues. And even if you don't use any new feature and keep the emulated machine, you may face issues - Paolo just recently fixed one that prevented starting a VM that was created in earlier RHEL version on RHEL 8.3 (the migration failed because of different CPU flags in QEMU). So if you really cares about stability, just don't make any change :) but realistically you should upgrade your hardware, both the physical and virtual one, and this means it exposes you to some level of risk. As a user this means that you should be careful when changing the cluster level and we provide users with the ability to examine their workloads on the new version.
(In reply to Arik from comment #28) > but at this point I really don't see a good reason to change it unless we > expect bunch of regressions by keep doing this. And then we'll need to ask ourselves why this happens now and didn't happen before...
(In reply to Arik from comment #28) > > Technically, it's not a big change but conceptually it is. > We do want our users to use the default settings in the cluster - that's > what we test and that brings great value in terms of maintaining a data > center over time. > I see that we now question this because we have this issue in the transition > from 8.3 to 8.4 and Igor clearly has a different view on it in terms of > stability vs. other criteria, > but at this point I really don't see a good reason to change it unless we > expect bunch of regressions by keep doing this. > The main thing here of course is that we don't really hit a bug in qemu, but we hit an issue in oVirt because we blindly (cause I think we don't check/announce in changelog) accept the changes being done in the emulate machine levels. If it was just a bug in Qemu, then I say 100%, things happen, fix this, and continue the way it works now. But we have the potential issue that when emulated machine 8.5.0 (or further) is the new default, we hit something else. And its not that it should not happen (from qemu point of view), the emulated machine version is bumped for a reason (breaking changes/new features). So I think its not a question of 'will this kind of bugs happen again', but more of 'when will it happen'. We come from 4.2, and upgraded to 4.6 without any issues except this one. But can we say when 4.7 is out its safe? Nope! > As for your proposal in comment 24 and the feedback on it in comment 25, I > agree with Igor that it may not solve everything and thus may not worth the > effort. > As mentioned before, even if you keep the emulated machine the same and > upgrade your hosts you get new features that may cause issues. And even if > you don't use any new feature and keep the emulated machine, you may face > issues - Paolo just recently fixed one that prevented starting a VM that was > created in earlier RHEL version on RHEL 8.3 (the migration failed because of > different CPU flags in QEMU). Agree, issues are always possible. But here we are talking about a breaking change by design (in qemu, which resulted in a different emulated machine version). > So if you really cares about stability, just > don't make any change :) but realistically you should upgrade your hardware, > both the physical and virtual one, and this means it exposes you to some > level of risk. As a user this means that you should be careful when changing > the cluster level and we provide users with the ability to examine their > workloads on the new version. That's the 'dream scenario' indeed. Unfortunately this is not always possible, surely not in a customer context. Some still run prehistoric OS'es etc, for whatever reasons :) And even if they had up-to-date OS'es, it would still give issues. Cause even an up-to-date Windows 2016 will hit this issue here. Take you have +100 Windows machines that will all lose their network because of this, you can't ask the customers to reconfigure all their NIC's.. We could do it, but for some machines we don't even have credentials to do it... So there is not always an easy solution.
(In reply to Jean-Louis Dupond from comment #30) > (In reply to Arik from comment #28) > > > > Technically, it's not a big change but conceptually it is. > > We do want our users to use the default settings in the cluster - that's > > what we test and that brings great value in terms of maintaining a data > > center over time. > > I see that we now question this because we have this issue in the transition > > from 8.3 to 8.4 and Igor clearly has a different view on it in terms of > > stability vs. other criteria, > > but at this point I really don't see a good reason to change it unless we > > expect bunch of regressions by keep doing this. > > > > The main thing here of course is that we don't really hit a bug in qemu, but > we hit an issue in oVirt because we blindly (cause I think we don't > check/announce in changelog) accept the changes being done in the emulate > machine levels. > If it was just a bug in Qemu, then I say 100%, things happen, fix this, and > continue the way it works now. > But we have the potential issue that when emulated machine 8.5.0 (or > further) is the new default, we hit something else. > And its not that it should not happen (from qemu point of view), the > emulated machine version is bumped for a reason (breaking changes/new > features). > So I think its not a question of 'will this kind of bugs happen again', but > more of 'when will it happen'. > We come from 4.2, and upgraded to 4.6 without any issues except this one. > But can we say when 4.7 is out its safe? Nope! Well, we do note the introduction of a new cluster level in the release notes and users upgrade their clusters manually, right? A different way to think about it is that it's not done "blindly" but the user chose not to pin the emulated machine of the VM to a particular one using a custom emulated machine, meaning in the oVirt terminology that the VM would use what the cluster is set with. And it's not a secret that each cluster-level is set with a different emulated machine. So oVirt users should really be aware of that, nothing is done "behind the scenes". It eventually comes to the question of whether users see the cluster upgrade as a confirmation for changing also existing VMs or not. I believe that's what we see differently - You say "you can't promise me that changing the cluster level won't damage my existing VM so I want it to run with its original emulated machine until I say otherwise and new VMs should be set with the latest emulated machine" While the existing approach assumes differently - that users make the relevant checks prior to upgrading the cluster and only then users proceed with the cluster upgrade. > > > > As for your proposal in comment 24 and the feedback on it in comment 25, I > > agree with Igor that it may not solve everything and thus may not worth the > > effort. > > As mentioned before, even if you keep the emulated machine the same and > > upgrade your hosts you get new features that may cause issues. And even if > > you don't use any new feature and keep the emulated machine, you may face > > issues - Paolo just recently fixed one that prevented starting a VM that was > > created in earlier RHEL version on RHEL 8.3 (the migration failed because of > > different CPU flags in QEMU). > > Agree, issues are always possible. But here we are talking about a breaking > change by design (in qemu, which resulted in a different emulated machine > version). But you chose to upgrade the cluster to a more advanced cluster-level, right? really, if you don't want to change the hardware of your workloads, don't do that.. > > > So if you really cares about stability, just > > don't make any change :) but realistically you should upgrade your hardware, > > both the physical and virtual one, and this means it exposes you to some > > level of risk. As a user this means that you should be careful when changing > > the cluster level and we provide users with the ability to examine their > > workloads on the new version. > > That's the 'dream scenario' indeed. Unfortunately this is not always > possible, surely not in a customer context. > Some still run prehistoric OS'es etc, for whatever reasons :) > And even if they had up-to-date OS'es, it would still give issues. Cause > even an up-to-date Windows 2016 will hit this issue here. > Take you have +100 Windows machines that will all lose their network because > of this, you can't ask the customers to reconfigure all their NIC's.. > We could do it, but for some machines we don't even have credentials to do > it... So there is not always an easy solution. What I mean here is that regardless of the operating system and process you run within the guest, you can test how it work prior to upgrading the cluster using the custom emulated machine and custom compatibility version.
(In reply to Arik from comment #31) > (In reply to Jean-Louis Dupond from comment #30) > > (In reply to Arik from comment #28) > > > > > > Technically, it's not a big change but conceptually it is. > > > We do want our users to use the default settings in the cluster - that's > > > what we test and that brings great value in terms of maintaining a data > > > center over time. > > > I see that we now question this because we have this issue in the transition > > > from 8.3 to 8.4 and Igor clearly has a different view on it in terms of > > > stability vs. other criteria, > > > but at this point I really don't see a good reason to change it unless we > > > expect bunch of regressions by keep doing this. > > > > > > > The main thing here of course is that we don't really hit a bug in qemu, but > > we hit an issue in oVirt because we blindly (cause I think we don't > > check/announce in changelog) accept the changes being done in the emulate > > machine levels. > > If it was just a bug in Qemu, then I say 100%, things happen, fix this, and > > continue the way it works now. > > But we have the potential issue that when emulated machine 8.5.0 (or > > further) is the new default, we hit something else. > > And its not that it should not happen (from qemu point of view), the > > emulated machine version is bumped for a reason (breaking changes/new > > features). > > So I think its not a question of 'will this kind of bugs happen again', but > > more of 'when will it happen'. > > We come from 4.2, and upgraded to 4.6 without any issues except this one. > > But can we say when 4.7 is out its safe? Nope! > > Well, we do note the introduction of a new cluster level in the release > notes and users upgrade their clusters manually, right? > A different way to think about it is that it's not done "blindly" but the > user chose not to pin the emulated machine of the VM to a particular one > using a custom emulated machine, meaning in the oVirt terminology that the > VM would use what the cluster is set with. > And it's not a secret that each cluster-level is set with a different > emulated machine. > So oVirt users should really be aware of that, nothing is done "behind the > scenes". > > It eventually comes to the question of whether users see the cluster upgrade > as a confirmation for changing also existing VMs or not. > I believe that's what we see differently - > You say "you can't promise me that changing the cluster level won't damage > my existing VM so I want it to run with its original emulated machine until > I say otherwise and new VMs should be set with the latest emulated machine" > While the existing approach assumes differently - that users make the > relevant checks prior to upgrading the cluster and only then users proceed > with the cluster upgrade. > > > > > > > > As for your proposal in comment 24 and the feedback on it in comment 25, I > > > agree with Igor that it may not solve everything and thus may not worth the > > > effort. > > > As mentioned before, even if you keep the emulated machine the same and > > > upgrade your hosts you get new features that may cause issues. And even if > > > you don't use any new feature and keep the emulated machine, you may face > > > issues - Paolo just recently fixed one that prevented starting a VM that was > > > created in earlier RHEL version on RHEL 8.3 (the migration failed because of > > > different CPU flags in QEMU). > > > > Agree, issues are always possible. But here we are talking about a breaking > > change by design (in qemu, which resulted in a different emulated machine > > version). > > But you chose to upgrade the cluster to a more advanced cluster-level, > right? really, if you don't want to change the hardware of your workloads, > don't do that.. > > > > > > So if you really cares about stability, just > > > don't make any change :) but realistically you should upgrade your hardware, > > > both the physical and virtual one, and this means it exposes you to some > > > level of risk. As a user this means that you should be careful when changing > > > the cluster level and we provide users with the ability to examine their > > > workloads on the new version. > > > > That's the 'dream scenario' indeed. Unfortunately this is not always > > possible, surely not in a customer context. > > Some still run prehistoric OS'es etc, for whatever reasons :) > > And even if they had up-to-date OS'es, it would still give issues. Cause > > even an up-to-date Windows 2016 will hit this issue here. > > Take you have +100 Windows machines that will all lose their network because > > of this, you can't ask the customers to reconfigure all their NIC's.. > > We could do it, but for some machines we don't even have credentials to do > > it... So there is not always an easy solution. > > What I mean here is that regardless of the operating system and process you > run within the guest, you can test how it work prior to upgrading the > cluster using the custom emulated machine and custom compatibility version. All you say above applies to private cloud only, where owner of cluster and VMs is the same. It just doesn't work for public cloud, where VMs users have not clue what virt stack is used by hosting. Even for private cloud it's taxing procedure to upgrade cluster level, basically it's impossible to click button for cluster level upgrade, without manually checking and upgrading every existing VM, only after that it would be reasonably safe to upgrade cluster level.
(In reply to Igor Mammedov from comment #33) > All you say above applies to private cloud only, where owner of cluster and > VMs > is the same. It just doesn't work for public cloud, where VMs users have not > clue > what virt stack is used by hosting. > Even for private cloud it's taxing procedure to upgrade cluster level, > basically it's impossible to click button for cluster level upgrade, > without manually checking and upgrading every existing VM, only after > that it would be reasonably safe to upgrade cluster level. Yeah, as I wrote on comment 21, it might be different when it comes to clouds. I'm only speaking about oVirt and similar virtualization platforms for pet VMs here.
This bug diverged into a discussion about pinning VMs to the emulated machine they were created with. We can't avoid changing the emulated machine when upgrading the cluster-level in oVirt/RHV since it's a major conceptual change with many implications. Note that even by doing that, we wouldn't be able to completely eliminate the need to change the virtual hardware since there's no commitment from the QEMU side to support all emulated machines that were introduced in the past (e.g., QEMU on RHEL 8 doesn't support emulated machines of RHEL 6). So as mentioned earlier on the thread, this means this approach would only defer the changes to the virtual hardware, and as a result the possibility of having such issues, to a future major release of RHEL. One can currently achieve this by setting custom emulated machines when creating VMs. If one would like VMs to be set with a custom emulated machine automatically when they're created, one would need to contribute this as a cluster policy that can replace the default one. It's not something we plan to invest in. As for the original issue reported in this bz, based on comment 10 and comment 11, closing it as a duplicate of bz 1939546. *** This bug has been marked as a duplicate of bug 1939546 ***