In order to migrate big (3-6TB) VMs, the following virsh options have to be able to be specified in a migration policy: --parallel --parallel-connections <NUM> Either this gets added to the existing policies, e.g. "Minimal Downtime", or a new policy, e.g. "Migrate High Performance VM" would be required, too.
So that's one option that should help and we should make available to people; picking the 'num' is a bit of guess work, it varies a lot depending on CPU, configuration and network bandwidth available.
since we're limiting the amount of concurrent migrations to a low number then enabling multifd might be ok by default, I can't imagine it would create too much overhead Interesting question would be what's the right number of fds. Naively I would think that it may be good enough to make it proportional to bandwidth only. Perhaps something simple as a 1 fd per 10 GBps.
(In reply to Michal Skrivanek from comment #3) Nils, what's your suggestion/experience with these VMs?
It depends on what the scarce resource it - either CPU or network. Given enough network, then it might be best to be CPU correlated. If network is limited, then we might want to correlated it to network.
So far most of my tests have been using --parallel-connections 2 which seemed okay, even on a 40G link, but network was not the bottleneck. Bandwidth usage was most of the time < 10G (around 1-2G) but with spikes in the 20G range. I like the KISS (Keep It Simple and Stupid) approach of having 1 fd per 10G for an automated guessing. There should be also be the way to specify an exact number, e.g. 2 in my case. So in the migration policy we could specify: parallelMigrations: false|true parallelMigrationConnections: auto|<NUM> Makes sense?
(In reply to Nils Koenig from comment #6) > There should be also be the way to specify an exact number, e.g. 2 in my > case. > > So in the migration policy we could specify: well, so that's the thing, if we use migration policies then they are quite cumbersome to edit, so it would be by far more easy to bake it into the policies by default. Would be best one-dimensional i.e. fdsPer10Gb=1 or the other way around perhaps, addFdPerGb=10. Changing would be via engine-config
I'd be careful when automating this; if someone has 200Gbp NIC, do you really want to automatically eat 20 threads during migration? It might be OK on huge boxes like Nils has, but a more modest box with a fast NIC might have problems, especially if they're doing parallel migrations and are running their CPUs close to the limit anyway on guest vCPUs.
(In reply to Dr. David Alan Gilbert from comment #8) > I'd be careful when automating this; if someone has 200Gbp NIC, do you > really want to automatically eat 20 threads during migration? > It might be OK on huge boxes like Nils has, but a more modest box with a > fast NIC might have problems, especially if they're doing parallel > migrations and are running their > CPUs close to the limit anyway on guest vCPUs. +1 I also like the KISS principle but I'm a bit concerned about the implication of simplifying it too much. I'd imaging it to be another property on the cluster - like "Migration bandwidth limit" with a configurable value for the amount of connections. As I don't suppose that 3-6 TB VMs are so common on RHV, I'd default it to 1 as we have today and users can override this value for VMs for which it makes sense. Maybe we can think of overriding that value by default based on the amount of RAM of the VM, need to think about it. I also think it could affect the scheduling as we'd like to pick a host that is able, CPU-wise and bandwidth-wise, to process the incoming migration.
Based on the discussions here and elsewhere, there are several things to take into account. First, there are resource limitations: - Network: Its capacity is obviously important, but it may not be obvious how much actual capacity is available. Even with dedicated migration networks and not considering what happens at switches etc. things may not be that easy because there can be multiple outgoing and incoming migrations, starting or finishing anytime. Would it make sense limiting the number of active migration connections rather than number of migrations? - CPU: If the source host is busy, it may not be the best to make it busy even more. There can be also special considerations such as NUMA and especially (future) dedicated CPUs. The destination host should always have free CPUs for the VM so as long as the number of threads is not greater than the number of the vCPUs it shouldn't be a problem I guess (which frees us from examining the situation on the destination). Except for post-copy, but in such a case we are motivated to migrate as fast as possible for good performance anyway. - Memory: This is not a limitation but perhaps larger VMs would benefit more from the parallel connections and thus RAM used (and maybe use of hugepages) could influence whether and how many connections to use. Second, making it meaningful for the users: - It should be simple, preferably mostly automatic. - We still need at least an option to disable/enable parallel connections, as a safety/troubleshoot switch if nothing else. - There is a danger that if it doesn't work very well initially, users will switch the feature off and will never enable it again. We don't want to deter users from the feature. - A serious complication is that the benefit or harm of a given number of parallel connections may be dependent on the current state of the system. Making a trivial rule is tempting but not necessarily very useful. The cluster and VM limits suggested by Arik in the preceding comment look like a good and safe start. Or there could be a simple hard limit in config, automatically adjusted per migration by selecting the number of connections based on the limit itself, the nominal network capacity, the number of current outgoing and incoming migration connections, CPU and RAM usage. Presenting this value as a changeable default in the migration dialog would allow the users to play with it to find out what works best for them but OTOH making the number so clearly visible might make them always unhappy and/or confused. What other ideas do you have?
(In reply to Milan Zamazal from comment #10) > - CPU: If the source host is busy, it may not be the best to make it busy > even more. Or it can, by slowing down the guest + making the migration faster. But I suppose that would work only when the host was mostly occupied by only the given VM.
(In reply to Milan Zamazal from comment #10) > Based on the discussions here and elsewhere, there are several things to > take into account. > > First, there are resource limitations: > > - Network: Its capacity is obviously important, but it may not be obvious > how much actual capacity is available. Even with dedicated migration > networks and not considering what happens at switches etc. things may not be > that easy because there can be multiple outgoing and incoming migrations, > starting or finishing anytime. Would it make sense limiting the number of > active migration connections rather than number of migrations? > - CPU: If the source host is busy, it may not be the best to make it busy > even more. There can be also special considerations such as NUMA and > especially (future) dedicated CPUs. The destination host should always have > free CPUs for the VM so as long as the number of threads is not greater than > the number of the vCPUs it shouldn't be a problem I guess (which frees us > from examining the situation on the destination). Except for post-copy, but > in such a case we are motivated to migrate as fast as possible for good > performance anyway. > - Memory: This is not a limitation but perhaps larger VMs would benefit more > from the parallel connections and thus RAM used (and maybe use of hugepages) > could influence whether and how many connections to use. > > Second, making it meaningful for the users: > > - It should be simple, preferably mostly automatic. > - We still need at least an option to disable/enable parallel connections, > as a safety/troubleshoot switch if nothing else. > - There is a danger that if it doesn't work very well initially, users will > switch the feature off and will never enable it again. We don't want to > deter users from the feature. > - A serious complication is that the benefit or harm of a given number of > parallel connections may be dependent on the current state of the system. > > Making a trivial rule is tempting but not necessarily very useful. The > cluster and VM limits suggested by Arik in the preceding comment look like a > good and safe start. Or there could be a simple hard limit in config, > automatically adjusted per migration by selecting the number of connections > based on the limit itself, the nominal network capacity, the number of > current outgoing and incoming migration connections, CPU and RAM usage. > Presenting this value as a changeable default in the migration dialog would > allow the users to play with it to find out what works best for them but > OTOH making the number so clearly visible might make them always unhappy > and/or confused. What other ideas do you have? From a user perspective I like when there are reasonable default settings chosen for me that work in most of the cases. On the other hand, I like when there is an "advanced" tab/menu where I can modify such parameters as: autoconverge, migration speed, migration parallelization (threads and number of parallel migrated VMs), migration strategy (pre-/ postcopy) The current migration policies do not expose all of the settings to the end user, e.g. autoconverge. I think giving the user the ability to modify those settings if they have to in an easy (GUI) manner, makes sense from my point of view.
(In reply to Nils Koenig from comment #12) > autoconverge, > migration speed, > migration parallelization (threads and number of parallel migrated VMs), > migration strategy (pre-/ postcopy) > > > The current migration policies do not expose all of the settings to the end > user, e.g. autoconverge. > I think giving the user the ability to modify those settings if they have to > in an easy (GUI) manner, makes sense from my point of view. This couples the amount of connections to the migration policy and I don't see why In my opinion, the amount of parallel connection is similar to 'migration bandwidth' that is defined outside of the migration policy I change the title not to impose my view but since the requirement here is about having this parameter - whether it would be part of the migration policies or not is an implenetation detail
(In reply to Milan Zamazal from comment #10) > - Network: Its capacity is obviously important, but it may not be obvious > how much actual capacity is available. Even with dedicated migration > networks and not considering what happens at switches etc. things may not be > that easy because there can be multiple outgoing and incoming migrations, > starting or finishing anytime. Would it make sense limiting the number of > active migration connections rather than number of migrations? I think so > - We still need at least an option to disable/enable parallel connections, > as a safety/troubleshoot switch if nothing else. We'll need a FeatureSupported field for this since I see the work on that in libvirt was done around Feb 2019 so it should be disabled for 4.2 cluster. However, it would probably be a good idea to enable it only on the last cluster-level anyway to reduce the amount of testing. But in addition I think it would be nice to have a cluster-level property (like the migration bandwidth) that can be overridden per VM (like migration encryption) This would enable users to enable/disable it easily per cluster (assuming it has not been overridden for specific VMs) > What other ideas do you have? I think you summarized it well I imagine a field that is set at the cluster level and can be overridden per VM with something like the following options: AUTO (default) - the system picks the right amount of connections automatically CUSTOM - the user specifies the amount of connection DISABLED - only a single connection is used And the challenge would be to find a reasonable computation for 'AUTO' when the trade off is between enabling it more widely (and then it will be used more often but it's also more risky) and more specifically (and then it's less used even when it can help but it's safer). I wouldn't expose things in the migration dialog but rather in the add/edit vm dialog since many times the migration is initiated by the system and adding it to the 'migrate' verb in the API is not a good idea but having them as part of the vm configuration makes it less likely that users will tune them. If we come up with few properties that can be adjusted using engine-config then I think it would be good enough
> I imagine a field that is set at the cluster level and can be overridden per VM with something like the following options: > AUTO (default) - the system picks the right amount of connections automatically > CUSTOM - the user specifies the amount of connection > DISABLED - only a single connection is used Sounds good to me, this way it would work automatically by default and the user would still have all basic means to override it. > And the challenge would be to find a reasonable computation for 'AUTO' when the trade off is between enabling it more widely (and then it will be used more often but it's also more risky) and more specifically (and then it's less used even when it can help but it's safer). How about, based on previous comments, starting with min(bandwith / 10 Gbps, current_number_of_vm_vcpus)? Can anybody see a problem with such a simple heuristics? > I wouldn't expose things in the migration dialog but rather in the add/edit vm dialog since many times the migration is initiated by the system Ah, right. > and adding it to the 'migrate' verb in the API is not a good idea but having them as part of the vm configuration makes it less likely that users will tune them. I'm not sure I understand here -- the eventual number of the parallel connections must be passed to VM.migrate, or? Also, there is still the question whether the number of parallel connections should be counted against the number of outgoing/incoming migrations (and the limit should be adjusted in such a case) or not. If we are going to consider the network bandwidth then probably not. And the current limit of 2 is good for ensuring a simple rule such as the heuristics suggested above doesn't cause overloading.
(In reply to Milan Zamazal from comment #15) > > And the challenge would be to find a reasonable computation for 'AUTO' when the trade off is between enabling it more widely (and then it will be used more often but it's also more risky) and more specifically (and then it's less used even when it can help but it's safer). > > How about, based on previous comments, starting with min(bandwith / 10 Gbps, > current_number_of_vm_vcpus)? Can anybody see a problem with such a simple > heuristics? +1 > > and adding it to the 'migrate' verb in the API is not a good idea but having them as part of the vm configuration makes it less likely that users will tune them. > > I'm not sure I understand here -- the eventual number of the parallel > connections must be passed to VM.migrate, or? Ah I referred to REST-API, not the API between ovirt-engine and VDSM - the number of parallel connection surely needs to be added to the latter > If we are going to consider the network bandwidth then probably not. And the current limit of 2 > is good for ensuring a simple rule such as the heuristics suggested above > doesn't cause overloading. Yeah, makes sense to me to keep the limit of 2 then
Milan, https://www.ovirt.org/develop/release-management/features/virt/parallel-migration-connections.html points to a developer related documentation. Can you please help reviewing https://github.com/oVirt/ovirt-site/pull/2828/files so we can get proper user documentation to point to?
(In reply to Sandro Bonazzola from comment #19) > > Can you please help reviewing > https://github.com/oVirt/ovirt-site/pull/2828/files so we can get proper > user documentation to point to? I've already approved the PR a week ago, I don't know why GitHub doesn't show it as approved by me. Approved again.
Verified with: ovirt-engine-4.5.0.2-0.7.el8ev.noarch vdsm-4.50.0.12-1.el8ev.x86_64 libvirt-8.0.0-5.module+el8.6.0+14480+c0a3aa0f.x86_64 Steps: Run the cases in the attached Polarion test plan(Only test Web UI, there is another bug tracking REST API part, see https://bugzilla.redhat.com/show_bug.cgi?id=2078500) Results: All cases passed.
This bugzilla is included in oVirt 4.5.0 release, published on April 20th 2022. Since the problem described in this bug report should be resolved in oVirt 4.5.0 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.