See also: https://bugzilla.redhat.com/show_bug.cgi?id=1810605

I can't reproduce the bug, a HP VM (just set as HP, without any further arrangements) migrates fine for me.

Polina, what's your systemd version?

I'm also confused about the environment description above. Do you mean "TSC" rather than "TCS"? If not, could you explain to me what is TCS? If yes, I can see "tsc" in your (and also my) CPU flags -- but you say it doesn't support TSC. Could you please clarify what you mean?

I upgraded my hosts to systemd 239-29 etc. and I can reproduce the error now. Engine filters out the migration destination host because it "has a different TSC frequency". Indeed, Vdsm reports slightly different frequencies for each of the hosts. Unlike Polina's hosts, my hosts don't report the cgroup errors, so those errors are probably unrelated. I could migrate the same VM on my machines in both the directions yesterday, when one of the machines had systemd 239-28 and some recent updates missing and the other one had a much older system.

No idea now, the bug needs closer inspection.

Huh, what has removed the keywords?

Can you upload more recent sosreports? Or at least dmesg/the systemd journal from those hosts? With scaling, the exact frequency shouldn't matter as long as the CPUs are the same

Yash -

Briefly discussed on Monday, but Milan is reporting that this breaks after a package update from RHEL. Are we aware of any regressions or bugs around this? I briefly looked in Bugzilla, but the search function is not always the best

Milan, can you provide the versions used?

On one host:
kernel 4.18.0-147.0.3.el8_1 -> 4.18.0-147.8.1.el8_1
systemd 239-18.el8_1.4 -> 239-29.el8
qemu 4.2.0-16 -> 4.2.0-17
libvirt 6.0.0-15 -> 6.0.0-16

On the other host:
kernel 4.18.0-147.0.3.el8_1 -> 4.18.0-147.8.1.el8_1
systemd 239-18.el8_1.4 -> 239-28.el8
qemu 4.1.0-23 -> 4.2.0-17
libvirt 5.6.0-10 -> 6.0.0-16

>fails to migrate with error The host host_mixed_1 did not satisfy internal filter Migration-Tsc-Frequency because it has a different TSC Frequency.]
>libvirt.libvirtError: Requested operation is not valid: cgroup CPU controller is not mounted

This is almost certainly an issue caused by systemd or libvirt's use of it. Requesting additional analysis from libvirt team by way of needinfo.

(In reply to Jeff Nelson from comment #9)
> >fails to migrate with error The host host_mixed_1 did not satisfy internal filter Migration-Tsc-Frequency because it has a different TSC Frequency.]
> >libvirt.libvirtError: Requested operation is not valid: cgroup CPU controller is not mounted
> 
> This is almost certainly an issue caused by systemd or libvirt's use of it.
> Requesting additional analysis from libvirt team by way of needinfo.

Jirka, can you please have a look? Thanks. Is there any relation to rhel7 bug 1815572? Thanks.

The error

    Requested operation is not valid: cgroup CPU controller is not mounted

comes from virDomainSetSchedulerParameters and it is unrelated to the
migration and tracked by bug 1808940.

Apparently migration did not even start, because the engine decided there is
no host to migrate the VM to:

    Migration failed due to a failed validation: [Cannot migrate VM. There is
    no host that satisfies current scheduling constraints. See below for
    details:, The host host_mixed_1 did not satisfy internal filter
    Migration-Tsc-Frequency because it has a different TSC Frequency.]

so the question is what oVirt is doing to check the TSC frequencies? Does it
compare the //cpu/counter[@name='tsc'] elements from the capabilities XMLs? If
so, could you please share the counter elements from both the source and the
destination host with us?

Libvirt is probing the TSC frequency from KVM via KVM_GET_TSC_KHZ ioctl so
further questions about why these values differ on your hosts should probably
go to the KVM folks.

After inspecting my environment and old logs I can see the problem in my environment is that on one of the hosts the reported TSC frequency has changed from 2133408000 Hz to 2133406000 Hz (and there is no TSC frequency scaling now or before). I tried to reboot the host and now the reported TSC frequency is 2133407000 Hz. So the reported frequency is nondeterministic and slightly variable, which causes the migration problem.

Looking into linux/arch/x86/kernel/tsc.c and dmesg on my host, the TSC frequency value comes from calibration and the kernel tries to keep it within certain bounds of accuracy. It's possible to get an exact value from hardware info on modern Intel CPUs but in other cases it's only measurement. If my observations are correct, we can't expect to have exactly the same TSC frequency values in many cases, even on the same machine across reboots (as my host demonstrates).
Now the question is how to deal with that fact in migrations. Do I assume correctly that libvirt would reject the VM on the destination if the TSC frequency specified in the domain XML wasn't the same as on the host? And can slightly different TSC frequencies cause any harm to HP VMs? One very rude and ugly way to deal with that would be to tolerate slight differences in Engine (assuming it's OK for HP VMs) and replace the TSC frequency in libvirt hook. Another way would be to restrict migrations to hardware providing exact TSC frequency info, which is perhaps too restrictive and quite confusing. Maybe libvirt could provide some assistance, which might be the best solution.
What do you all think?

Yes, libvirt compares the TSC frequency in domain XML with the frequency
probed from the host and refuses to start the domain if they don't exactly
match. Unfortunately, changing the TSC frequency during migration is forbidden
too, libvirt explicitly checks the frequencies match in both the original and
updated domain definition (either supplied by a parameter to the migration API
or via a pre-migration hook). We need to check whether the strict match is
really necessary by trying to create a domain with a TSC frequency which
slightly differs from the host's frequency which does not support scaling.

Jiri, this doesn't quite seem right. Isn't the point of migration on hosts with support TSC scaling that we could try to synchronize the frequency as part of the migration from libvirt?

I believe we're all talking about hosts with *do not* support TSC scaling as
otherwise the host's TSC frequency is irrelevant. If the destination host
supports TSC scaling, libvirt naturally doesn't check whether the TSC
frequency from domain XML matches the one natively used by the host.

Yes, QE hosts and my hosts don't support TSC scaling. I've forgotten to mention in my Comment 13 that the problems outlined there concern hosts without TSC scaling.

Clearing my needinfo as Jeff answered the question in #c9

So how can we move forward? Looking into QEMU sources, QEMU doesn't seem to like TSC frequency mismatch. Would it be possible to perform "frequency scaling" in libvirt, i.e. if the host doesn't support TSC frequency scaling and the requested TSC frequency is only slightly different (maybe depending on some attribute), libvirt would use the host TSC frequency? Could it work with migrations?

Jirko, does it make sense or not? Did you have a better idea in Comment 14?

Marcelo, how do you think we should handle migration between two identical
hosts which do not support TSC scaling, but the TSC frequency probed by the
kernel differs a bit. Currently libvirt refuses to migrate a domain between
these two hosts because of TSC frequency mismatch.

See comment 12 to 14 for further details.

(In reply to Jiri Denemark from comment #20)
> Marcelo, how do you think we should handle migration between two identical
> hosts which do not support TSC scaling, but the TSC frequency probed by the
> kernel differs a bit. 

Jiri, is this system exposing invtsc to the guest ?

If it is not, then there is no need to block migration.

If it is, then TSC clock frequency should be quite similar (we can calculate 
an error bound if necessary and allow in that range): but i don't think this is
necessary, i doubt invtsc is exposed to the guest.

> Currently libvirt refuses to migrate a domain between
> these two hosts because of TSC frequency mismatch.
> 
> See comment 12 to 14 for further details.

If invtsc is not exposed to the guest, then kvmclock should be used, 
which handles different TSC frequency between source and destination.

Jirko, what do you think about Marcelo's response in Comment 21? If I understand it right, there is no need to block the migration in either case (unless the TSC frequencies differ significantly). So could libvirt be made more tolerant to slightly different TSC frequencies?

First, we need a complete domain XML to check whether invtsc is exposed to the
guest. If it isn't, we need to relax the check in libvirt. On the other hand,
if invtsc is exposed to the guest, we need (as mentioned in comment 14) to try
starting a domain with TSC frequency which slightly differs from the host to
see if QEMU is tolerant to such difference or not. If not, we'd need to change
both QEMU and libvirt to fix this.

Created attachment 1689947 [details]
Example domain XML with TSC

Attaching an example of a full domain XML of a HP oVirt VM with TSC. Please let me know if you need anything more from the oVirt side.

(In reply to Milan Zamazal from comment #12)
> After inspecting my environment and old logs I can see the problem in my
> environment is that on one of the hosts the reported TSC frequency has
> changed from 2133408000 Hz to 2133406000 Hz (and there is no TSC frequency
> scaling now or before). I tried to reboot the host and now the reported TSC
> frequency is 2133407000 Hz. So the reported frequency is nondeterministic
> and slightly variable, which causes the migration problem.

Jiri,

KVM_SET_TSC_KHZ supports an error of 250 ppm (see tsc_tolerance_ppm and 
adjust_tsc_khz in arch/x86/kvm/x86.c in the kernel source).

This is within the delta reported at comment 12.

Can that code be added to libvirt as well?

The domain XML shows invtsc is exposed to the guest.

Thanks Marcelo for the reference to the kernel code:

    /* tsc tolerance in parts per million - default to 1/2 of the NTP threshold */
    static u32 __read_mostly tsc_tolerance_ppm = 250;
    module_param(tsc_tolerance_ppm, uint, S_IRUGO | S_IWUSR);

    static u32 adjust_tsc_khz(u32 khz, s32 ppm)
    {
        u64 v = (u64)khz * (1000000 + ppm);
        do_div(v, 1000000);
        return v;
    }

    static int kvm_set_tsc_khz(struct kvm_vcpu *vcpu, u32 user_tsc_khz)
    {
        ...
        thresh_lo = adjust_tsc_khz(tsc_khz, -tsc_tolerance_ppm);
        thresh_hi = adjust_tsc_khz(tsc_khz, tsc_tolerance_ppm);
        if (user_tsc_khz < thresh_lo || user_tsc_khz > thresh_hi) {
            pr_debug("kvm: requested TSC rate %u falls outside tolerance [%u,%u]\n", user_tsc_khz, thresh_lo, thresh_hi);
            use_scaling = 1;
        }
        ...
    }

So for the host TSC frequency 2133408000 Hz a domain can request anything
within +/- 533 kHz of the host frequency. The only problem is that
tsc_tolerance_ppm is a parameter of the kvm module.

We have two options, either use the default value in libvirt and hope nobody
changed it on the host or we can try setting the frequency via KVM and check
the result. But looking at the kernel code I don't see how we could
detect that setting TSC failed because of unsupported TSC scaling rather than
some other reason.

Anyway that it seems the kernel is less strict and allows setting TSC
frequency which is greater than host TSC frequency even without TSC scaling:

    static int set_tsc_khz(struct kvm_vcpu *vcpu, u32 user_tsc_khz, bool scale)
    {
        ...
        /* TSC scaling supported? */
        if (!kvm_has_tsc_control) {
            if (user_tsc_khz > tsc_khz) {
                vcpu->arch.tsc_catchup = 1;
                vcpu->arch.tsc_always_catchup = 1;
                return 0;
            } else {
                pr_warn_ratelimited("user requested TSC rate below hardware speed\n");
                return -1;
            }
        }
        ...
    }

There's definitely something to be improved in libvirt.

I checked the code in QEMU and it calls KVM_SET_TSC_KHZ first and checks TSC
frequencies only if the call fails. In other words, libvirt is the only part
which needs fixing here because it is too strict.

Marcelo, it seems the behavior does not match how I would understand the code
in QEMU and the kernel (in comment 26). On a host without TSC scaling QEMU
fails to set the frequency unless it is exactly the same as reported by the
kernel:

From virsh capabilities:

    <counter name='tsc' frequency='2903993000' scaling='no'/>

TSC requested 1 kHz below the host:

    $ /usr/bin/qemu-system-x86_64 -machine pc,accel=kvm -cpu host,invtsc=on,tsc-frequency=2903992000
    qemu-system-x86_64: warning: TSC frequency mismatch between VM (2903992 kHz) and host (2903993 kHz), and TSC scaling unavailable
    qemu-system-x86_64: kvm_init_vcpu failed: Operation not supported

TSC requested 1 kHz above the host:

    $ /usr/bin/qemu-system-x86_64 -machine pc,accel=kvm -cpu host,invtsc=on,tsc-frequency=2903994000
    qemu-system-x86_64: warning: TSC frequency mismatch between VM (2903994 kHz) and host (2903993 kHz), and TSC scaling unavailable
    qemu-system-x86_64: kvm_init_vcpu failed: Operation not supported

Exact TSC:

    $ /usr/bin/qemu-system-x86_64 -machine pc,accel=kvm -cpu host,invtsc=on,tsc-frequency=2903993000
    # QEMU runs happily here

The TSC tolerance is the default value (which corresponds to +/- 726 kHz
interval around the host frequency on this particular host):

    # cat /sys/module/kvm/parameters/tsc_tolerance_ppm
    250

I tried this on several hosts without TSC scaling and the behavior is the same
everywhere. Did I misunderstand anything? Or am I just doing it all wrong?

(In reply to Jiri Denemark from comment #28)
> Marcelo, it seems the behavior does not match how I would understand the code
> in QEMU and the kernel (in comment 26). On a host without TSC scaling QEMU
> fails to set the frequency unless it is exactly the same as reported by the
> kernel:
> 
> From virsh capabilities:
> 
>     <counter name='tsc' frequency='2903993000' scaling='no'/>
> 
> TSC requested 1 kHz below the host:
> 
>     $ /usr/bin/qemu-system-x86_64 -machine pc,accel=kvm -cpu
> host,invtsc=on,tsc-frequency=2903992000
>     qemu-system-x86_64: warning: TSC frequency mismatch between VM (2903992
> kHz) and host (2903993 kHz), and TSC scaling unavailable
>     qemu-system-x86_64: kvm_init_vcpu failed: Operation not supported
> 
> TSC requested 1 kHz above the host:
> 
>     $ /usr/bin/qemu-system-x86_64 -machine pc,accel=kvm -cpu
> host,invtsc=on,tsc-frequency=2903994000
>     qemu-system-x86_64: warning: TSC frequency mismatch between VM (2903994
> kHz) and host (2903993 kHz), and TSC scaling unavailable
>     qemu-system-x86_64: kvm_init_vcpu failed: Operation not supported
> 
> Exact TSC:
> 
>     $ /usr/bin/qemu-system-x86_64 -machine pc,accel=kvm -cpu
> host,invtsc=on,tsc-frequency=2903993000
>     # QEMU runs happily here
> 
> The TSC tolerance is the default value (which corresponds to +/- 726 kHz
> interval around the host frequency on this particular host):
> 
>     # cat /sys/module/kvm/parameters/tsc_tolerance_ppm
>     250
> 
> I tried this on several hosts without TSC scaling and the behavior is the
> same
> everywhere. Did I misunderstand anything? Or am I just doing it all wrong?

Jiri,

Maybe i also misunderstood the code.

I'll confirm and get back to you (will submit a patch to change this upstream 
if necessary).

Per the current target milestone of bz 1839095, moving to ovirt-4.4.3

Marcelo, did you have a chance to look at it and did you find anything that would help us move on with this bug?

(In reply to Milan Zamazal from comment #31)
> Marcelo, did you have a chance to look at it and did you find anything that
> would help us move on with this bug?

Hi Milan,

A fix was posted upstream (to kvm and qemu). Will backport.

(In reply to Marcelo Tosatti from comment #32)

> A fix was posted upstream (to kvm and qemu). Will backport.

Hi Marcelo, excellent news, thank you! Is the fix tracked anywhere so that we can watch it or will you notify us here once the backport is available for us?

(In reply to Milan Zamazal from comment #33)
> (In reply to Marcelo Tosatti from comment #32)
> 
> > A fix was posted upstream (to kvm and qemu). Will backport.
> 
> Hi Marcelo, excellent news, thank you! Is the fix tracked anywhere so that
> we can watch it or will you notify us here once the backport is available
> for us?

Not really, just this BZ. I'll open separate tickets and mark this as dependent on them.

Hi Marcelo, are there any news regarding your patches or the tickets?

(In reply to Milan Zamazal from comment #35)
> Hi Marcelo, are there any news regarding your patches or the tickets?

Opened a BZ to track the backport for the kernel fix.

Bug https://bugzilla.redhat.com/show_bug.cgi?id=1882793 backported the kernel which should
hopefully fix this problem.

The fix for bz 1882793 is in RHEL 8.4

libvirt implemented a tolerance for TSC frequency difference on migrations, but it still reports the exact TSC frequency value as measured on each of the hosts. So we will need to implement a similar tolerance in Engine in order not to reject the migrations already there.

There is an additional problem, as discussed on the users list: When the TSC frequency check is disabled in the web UI, the VM still doesn't migrate. While libvirtVmXmlBuilder.java does the right job and doesn't set VM TSC frequency in such a case, the filter in MigrationTscFrequencyPolicyUnit.java seems to ignore the setting.

verified on hosts lynx09.lab.eng.tlv2.redhat.com - lynx11.lab.eng.tlv2.redhat.com  ( <counter name='tsc' frequency='2300026000' scaling='no'/>)
ovirt-engine-4.4.6.5-0.17.el8ev.noarch

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: RHV Manager security update (ovirt-engine) [ovirt-4.4.6]), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2179