Bug 584581

Summary: dom0_max_vcpus=2 is not enough to allow pinning dom0's VCPUs
Product: Red Hat Enterprise Linux 5 Reporter: Bill Braswell <bbraswel>
Component: kernel-xenAssignee: Paolo Bonzini <pbonzini>
Status: CLOSED NOTABUG QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 5.5CC: drjones, jbrier, jwest, mshao, pbonzini, xen-maint, yuzhang
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-04-05 15:49:02 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 514491    
Attachments:
Description Flags
patch to fix the bug none

Description Bill Braswell 2010-04-21 21:21:04 UTC 
Description of problem:
Xen 'xm vcpu-pin 0 0 0' does not work:

# /usr/sbin/xm vcpu-pin 0 0 0
Error: (22, 'Invalid argument')
Usage: xm vcpu-pin <Domain<VCPU<CPUs>

However, the customer tested upstream Xen 3.4.2 and it worked fine:

# /usr/sbin/xm vcpu-pin 0 0 0
# /usr/sbin/xm vcpu-pin 0 1 2
# /usr/sbin/xm vcpu-list
Name                                ID  VCPU   CPU State   Time(s) CPU
Affinity
Domain-0                             0     0     0   -b-    2293.2 0
Domain-0                             0     1     2   r--     492.6 2
 .
 .
 .

Version-Release number of selected component (if applicable):
5.5, 2.6.18-194.el5xen
Customer is able to reproduce the problem on a RHEL 5.4 xen system
Comment 2 Paolo Bonzini 2010-04-22 07:30:06 UTC 
Try booting with dom0_vcpus_pin=0, it was changed by

commit c41ed05dc7a3078fe2234dc8c65577c0695689c7
Author: Rik van Riel <riel>
Date:   Thu Dec 13 15:13:52 2007 -0500

    [xen] disable cpu freq scaling when vcpus is small
Comment 4 Paolo Bonzini 2010-04-28 11:28:45 UTC 
1) The option goes on the xen kernel line.  Sorry for not being more specific.


2) "small" means actually "not all of the host CPUs".  This can happen at boot time due to the "dom0_max_vcpus" option, or at runtime due to "xm vcpu-pin".  The hypervisor does not support disabling/enabling CPU frequency scaling at run-time, so it has to make a decision at boot time: whether to enable CPU frequency scaling, or whether to allow pinning dom0's VCPUs.

What the patch does then is the following:

- add a command line option dom0_vcpus_pin

- if the option is 1, disable dom0 VCPU pinning *and* enable CPU frequency scaling

The option's default value depending on "dom0_max_vcpus".  If it is not found or equal to the number of physical CPUs, dom0_vcpus_pin will be 1 by default.  If it is different from the number of physical CPUs (which typically means "small"), dom0_vcpus_pin will be 0 by default.

So, an alternative (and if applicable, better) fix will be to set dom0_max_vcpus in the Xen kernel line.


3) I agree that this should be documented.  The behavior changed in 5.2 by the way.
Comment 6 Paolo Bonzini 2010-04-29 16:32:09 UTC 
I must admit I wrote everything from reading the code rather than really trying it out.

This code that actually uses the value of opt_dom0_vcpus_pin

    if ( (domid == 0) && opt_dom0_vcpus_pin )
        d->is_pinned = 1;

runs after "Brought up 4 CPUs" is printed.  This is before the value of dom0_max_vcpus is taken into account (which is just after "Dom0 has maximum 2 VCPUs" is printed).

So indeed dom0_vcpus_pin=0 is necessary too, this is a minor bug.
Comment 8 Paolo Bonzini 2010-08-02 09:52:35 UTC 
I haven't really looked at it.  Since there is a workaround, we didn't give it a high priority.  However, it is a bug and it's going to be fixed, if not for 5.6.
Comment 11 Paolo Bonzini 2011-03-25 16:30:19 UTC 
Created attachment 487594 [details]
patch to fix the bug

One-line patch, will ask for exception