Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1509821

Summary: Should fail to start guest when set tsc-frequency less than 1000hz
Product: Red Hat Enterprise Linux 7 Reporter: yafu <yafu>
Component: qemu-kvm-rhevAssignee: Marcelo Tosatti <mtosatti>
Status: CLOSED NOTABUG QA Contact: Sitong Liu <siliu>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.5CC: chayang, dyuan, fjin, juzhang, knoel, michen, siliu, virt-maint, xfu, yafu, yalzhang, zpeng
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-07-10 12:28:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
trace none

Description yafu 2017-11-06 07:18:55 UTC
Description of problem:
Should fail to start guest when set tsc-frequency less than 1000hz.
 
Version-Release number of selected component (if applicable):
libvirt-3.9.0-1.el7.x86_64
kernel-3.10.0-765.el7.x86_64
qemu-kvm-rhev-2.10.0-4.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Prepare a domain xml with tsc-frequency less than 1000hz:
#virsh edit vm1
...
 <clock offset='utc'>
    <timer name='tsc' tickpolicy='catchup' frequency='100'/>
...
  </clock>
...

2.Start the domain:
#virsh start vm1
Domain vm1 started

3.Check the qemu cmd line:
#ps aux | grep -i frequency
qemu     188062 30.3  0.9 1840624 320832 ?      Sl   02:09   0:12 /usr/libexec/qemu-kvm -name guest=avocado-vt-vm1,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-32-avocado-vt-vm1/master-key.aes -machine pc-i440fx-rhel7.5.0,accel=kvm,usb=off,dump-guest-core=off -cpu Skylake-Client,ds=on,acpi=on,ss=on,ht=on,tm=on,pbe=on,dtes64=on,monitor=on,ds_cpl=on,vmx=on,smx=on,est=on,tm2=on,xtpr=on,pdcm=on,osxsave=on,tsc_adjust=on,pdpe1gb=on,mpx=off,xsavec=off,xgetbv1=off,****tsc-frequency=100**** -m 1024 ...

4.Prepare a domain xml with tsc-frequency more than 1000hz but not equal to the host frequency:
#virsh edit vm1
...
 <clock offset='utc'>
    <timer name='tsc' tickpolicy='catchup' frequency='10000'/>
...
  </clock>
...

5.Start the guest:
# virsh start vm1
error: Failed to start domain vm1
error: internal error: process exited while connecting to monitor: 2017-11-03T06:12:05.069586Z qemu-kvm: -chardev pty,id=charserial0: char device redirected to /dev/pts/1 (label charserial0)
warning: host doesn't support requested feature: CPUID.01H:EDX.ds [bit 21]
warning: host doesn't support requested feature: CPUID.01H:EDX.acpi [bit 22]
warning: host doesn't support requested feature: CPUID.01H:EDX.ht [bit 28]
warning: host doesn't support requested feature: CPUID.01H:EDX.tm [bit 29]
warning: host doesn't support requested feature: CPUID.01H:EDX.pbe [bit 31]
warning: host doesn't support requested feature: CPUID.01H:ECX.dtes64 [bit 2]
warning: host doesn't support requested feature: CPUID.01H:ECX.monitor [bit 3]
warning: host doesn't support requested feature: CPUID.01H:ECX.ds-cpl [bit 4]
warning: host doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]
warning: host doesn't support requested feature: CPUID.01H:ECX.smx [bit 6]
warning: host doesn't support requested feature: CPUID.01H:ECX.est [bit 7]
warning: host do

6.Check the error info in qemu.log:
#cat /var/log/libvirt/qemu/vm1.log
...
2017-11-03T06:12:05.078983Z qemu-kvm: warning: TSC frequency mismatch between VM (10 kHz) and host (2099998 kHz), and TSC scaling unavailable
kvm_init_vcpu failed: Operation not supported
2017-11-03 06:12:05.165+0000: shutting down, reason=failed
...

Actual results:
Guest start successfully with tsc-frequency less than 1000hz but start failed with tsc-frequency more than 1000hz, the action is inconsistent.

Expected results:
Should failed to start guest when set tsc-frequency less than 1000hz and report the same error info as in step6.

Comment 5 Marcelo Tosatti 2018-07-05 18:05:44 UTC
I can't see why vcpu->arch.virtual_tsc_khz is not being set to 10 kHz, when KVM_SET_TSC_KHZ is called from 

static int kvm_arch_set_tsc_khz(CPUState *cs)
{
    X86CPU *cpu = X86_CPU(cs);
    CPUX86State *env = &cpu->env;
    int r;

    if (!env->tsc_khz) {
        return 0;
    }

    r = kvm_check_extension(cs->kvm_state, KVM_CAP_TSC_CONTROL) ?
        kvm_vcpu_ioctl(cs, KVM_SET_TSC_KHZ, env->tsc_khz) :
        -ENOTSUP;
    if (r < 0) {
        /* When KVM_SET_TSC_KHZ fails, it's an error only if the current
         * TSC frequency doesn't match the one we want.
         */

        int cur_freq = kvm_check_extension(cs->kvm_state, KVM_CAP_GET_TSC_KHZ) ?
                       kvm_vcpu_ioctl(cs, KVM_GET_TSC_KHZ) :
                       -ENOTSUP;
        if (cur_freq <= 0 || cur_freq != env->tsc_khz) {
            warn_report("TSC frequency mismatch between "
                        "VM (%" PRId64 " kHz) and host (%d kHz), "
                        "and TSC scaling unavailable",
                        env->tsc_khz, cur_freq);
            return r;
        }
    }

    return 0;
}


Can you please try this debug kernel (installed in the host),
and reproduce the problem (for the case which fails):
 
https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=17010006

And then copy /sys/kernel/debug/tracing/trace to a file, and attach 
to the BZ?

Thanks

Comment 6 yafu 2018-07-06 07:45:01 UTC
Created attachment 1456885 [details]
trace

Comment 7 Marcelo Tosatti 2018-07-06 16:03:27 UTC
Ok, so KVM_SET_TSC_KHZ is not even being called.

Can you please download the following script:

https://git.qemu.org/?p=qemu.git;a=blob_plain;f=scripts/kvm/vmxcap;hb=HEAD

Run it on the host system in question, and attach the output here?

Thanks

Comment 8 yafu 2018-07-09 02:22:17 UTC
(In reply to Marcelo Tosatti from comment #7)
> Ok, so KVM_SET_TSC_KHZ is not even being called.
> 
> Can you please download the following script:
> 
> https://git.qemu.org/?p=qemu.git;a=blob_plain;f=scripts/kvm/vmxcap;hb=HEAD
> 
> Run it on the host system in question, and attach the output here?
> 
> Thanks

# python tsc.py 
Basic VMX Information
  Hex: 0xda040000000012
  Revision                                 18
  VMCS size                                1024
  VMCS restricted to 32 bit addresses      no
  Dual-monitor support                     yes
  VMCS memory type                         6
  INS/OUTS instruction information         yes
  IA32_VMX_TRUE_*_CTLS support             yes
pin-based controls
  External interrupt exiting               yes
  NMI exiting                              yes
  Virtual NMIs                             yes
  Activate VMX-preemption timer            yes
  Process posted interrupts                yes
primary processor-based controls
  Interrupt window exiting                 yes
  Use TSC offsetting                       yes
  HLT exiting                              yes
  INVLPG exiting                           yes
  MWAIT exiting                            yes
  RDPMC exiting                            yes
  RDTSC exiting                            yes
  CR3-load exiting                         default
  CR3-store exiting                        default
  CR8-load exiting                         yes
  CR8-store exiting                        yes
  Use TPR shadow                           yes
  NMI-window exiting                       yes
  MOV-DR exiting                           yes
  Unconditional I/O exiting                yes
  Use I/O bitmaps                          yes
  Monitor trap flag                        yes
  Use MSR bitmaps                          yes
  MONITOR exiting                          yes
  PAUSE exiting                            yes
  Activate secondary control               yes
secondary processor-based controls
  Virtualize APIC accesses                 yes
  Enable EPT                               yes
  Descriptor-table exiting                 yes
  Enable RDTSCP                            yes
  Virtualize x2APIC mode                   yes
  Enable VPID                              yes
  WBINVD exiting                           yes
  Unrestricted guest                       yes
  APIC register emulation                  yes
  Virtual interrupt delivery               yes
  PAUSE-loop exiting                       yes
  RDRAND exiting                           yes
  Enable INVPCID                           yes
  Enable VM functions                      yes
  VMCS shadowing                           yes
  Enable ENCLS exiting                     no
  RDSEED exiting                           yes
  Enable PML                               yes
  EPT-violation #VE                        yes
  Conceal non-root operation from PT       no
  Enable XSAVES/XRSTORS                    no
  Mode-based execute control (XS/XU)       no
  TSC scaling                              no
VM-Exit controls
  Save debug controls                      default
  Host address-space size                  yes
  Load IA32_PERF_GLOBAL_CTRL               yes
  Acknowledge interrupt on exit            yes
  Save IA32_PAT                            yes
  Load IA32_PAT                            yes
  Save IA32_EFER                           yes
  Load IA32_EFER                           yes
  Save VMX-preemption timer value          yes
  Clear IA32_BNDCFGS                       no
  Conceal VM exits from PT                 no
VM-Entry controls
  Load debug controls                      default
  IA-32e mode guest                        yes
  Entry to SMM                             yes
  Deactivate dual-monitor treatment        yes
  Load IA32_PERF_GLOBAL_CTRL               yes
  Load IA32_PAT                            yes
  Load IA32_EFER                           yes
  Load IA32_BNDCFGS                        no
  Conceal VM entries from PT               no
Miscellaneous data
  Hex: 0x300481e5
  VMX-preemption timer scale (log2)        5
  Store EFER.LMA into IA-32e mode guest control yes
  HLT activity state                       yes
  Shutdown activity state                  yes
  Wait-for-SIPI activity state             yes
  IA32_SMBASE support                      yes
  Number of CR3-target values              4
  MSR-load/store count recommendation      0
  IA32_SMM_MONITOR_CTL[2] can be set to 1  yes
  VMWRITE to VM-exit information fields    yes
  Inject event with insn length=0          no
  MSEG revision identifier                 0
VPID and EPT capabilities
  Hex: 0xf0106334141
  Execute-only EPT translations            yes
  Page-walk length 4                       yes
  Paging-structure memory type UC          yes
  Paging-structure memory type WB          yes
  2MB EPT pages                            yes
  1GB EPT pages                            yes
  INVEPT supported                         yes
  EPT accessed and dirty flags             yes
  Single-context INVEPT                    yes
  All-context INVEPT                       yes
  INVVPID supported                        yes
  Individual-address INVVPID               yes
  Single-context INVVPID                   yes
  All-context INVVPID                      yes
  Single-context-retaining-globals INVVPID yes
VM Functions
  Hex: 0x1
  EPTP Switching                           yes

Comment 9 Marcelo Tosatti 2018-07-10 12:28:14 UTC
(In reply to yafu from comment #8)
> (In reply to Marcelo Tosatti from comment #7)

>   TSC scaling                              no

This machine does not have TSC scaling.

So the fact that emulating higher TSC frequencies than the host 
(which is called "TSC catchup" feature) is allowed, 
and lower TSC frequencies than the host are not allowed, is expected.

So closing this as not a bug.