Bug 876415

Summary: virDomainGetVcpuPinInfo might fail to show right CPU affinity setting
Product: Red Hat Enterprise Linux 6 Reporter: Wayne Sun <gsun>
Component: libvirtAssignee: Gunannan Ren <gren>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.4CC: acathrow, dallan, dyasny, dyuan, honzhang, jyang, mzhan, rwu
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-0.10.2-11.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-02-21 07:26:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 886216    

Description Wayne Sun 2012-11-14 03:35:41 UTC
Description
virDomainGetVcpuPinInfo can't show right CPU affinity setting

Version
libvirt-0.10.2-8.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.317.el6.x86_64
kernel-2.6.32-339.el6.x86_64

How reproducible:
sometime


Steps
1.
# virsh list
 Id    Name                           State
----------------------------------------------------
 3     libvirt_test_api               running

# virsh dumpxml libvirt_test_api |grep vcpu
  <vcpu placement='static'>4</vcpu>


2.
# python
Python 2.6.6 (r266:84292, Aug 28 2012, 10:55:56)
[GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>  import libvirt
>>>  conn = libvirt.open(None)
>>>  domobj = conn.lookupByName('libvirt_test_api')
>>>  domobj.vcpus()
([(0, 1, 8920000000L, 24), (1, 1, 3120000000L, 24), (2, 1, 3440000000L, 26), (3, 1, 3360000000L, 9)], [(True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True), (True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True), (True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True), (True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True)])
>>>  domobj.pinVcpuFlags(0, (True, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False), 1)
0
>>>  domobj.vcpus()
([(0, 1, 8960000000L, 0), (1, 1, 3170000000L, 8), (2, 1, 3500000000L, 26), (3, 1, 3410000000L, 8)], [(True, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False), (True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True), (True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True), (True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True)])
>>>  domobj.vcpuPinInfo(1)
[(True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True), (True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True), (True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True), (True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True)]
>>>


vcpuPinInfo shows vcpu 0 not pin to cpu 0 but all available cpus.

Check with virsh commands:
# virsh vcpuinfo libvirt_test_api
VCPU:           0
CPU:            0
State:          running
CPU time:       9.0s
CPU Affinity:   y-------------------------------

VCPU:           1
CPU:            8
State:          running
CPU time:       3.2s
CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

VCPU:           2
CPU:            10
State:          running
CPU time:       3.5s
CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

VCPU:           3
CPU:            24
State:          running
CPU time:       3.4s
CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

3.


Actual result
vcpuPinInfo not working


Expect result
vcpuPinInfo should working

Additional info

virsh vcpuinfo by default use virDomainGetVcpus, just fallback to use virDomainGetVcpuPinInfo if virDomainGetVcpus fails.

This problem can't be reproduced every time, but always on two ibm-x3850x5 boxes, this might not be related to hardware, maybe inconsistent data internally.

Comment 2 Gunannan Ren 2012-11-20 11:34:00 UTC
This bug only could be reproduced on machine with a big number of cpus(32 in this case)
virsh subcommand vcpuinfo works fine, only python API vcpuPinInfo() reports wrong informations.

Comment 3 Gunannan Ren 2012-11-23 07:01:42 UTC
This bug is caused by a typo of using wrong integer constant type 1U, it should use 1UL instead. So this leads to the weird situation when the total number of physical cpus on host is relatively big such as 32 but less than 64, incorrect vcpupin information will be output via virDomainGetVcpuPinInfo() API for qemu-kvm hypervisor.

Patch sent out to upstream.
https://www.redhat.com/archives/libvir-list/2012-November/msg01025.html

Comment 4 Gunannan Ren 2012-11-28 11:20:35 UTC
commit 237629d204b0674885a18ae10326938603b214d4
Author: Guannan Ren <gren>
Date:   Wed Nov 28 14:15:38 2012 +0800

    bitmap: fix typo to use UL type of integer constant in virBitmapIsAllSet
    
    This bug leads to getting incorrect vcpupin information via
    qemudDomainGetVcpuPinInfo() API when the number of maximum
    cpu on a host falls into a range such as 31 < ncpus < 64.
    
    gcc warning:
    left shift count >= width of type
    
    The following bug is such the case
    https://bugzilla.redhat.com/show_bug.cgi?id=876415

Patch send-out
http://post-office.corp.redhat.com/archives/rhvirt-patches/2012-November/msg00309.html

Move to POST

Comment 6 Wayne Sun 2012-12-06 04:26:45 UTC
pkgs:
# rpm -q libvirt libvirt-python qemu-kvm-rhev kernel
libvirt-0.10.2-11.el6.x86_64
libvirt-python-0.10.2-11.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.337.el6.x86_64
kernel-2.6.32-341.el6.x86_64

steps:
on a ibm x3850x5 box with 32 cpus:
# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                32
On-line CPU(s) list:   0-31
Thread(s) per core:    2
Core(s) per socket:    8
CPU socket(s):         2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 47
Stepping:              2
CPU MHz:               1064.000
BogoMIPS:              3989.86
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              18432K
NUMA node0 CPU(s):     0-7,16-23
NUMA node1 CPU(s):     8-15,24-31

# virsh list
 Id    Name                           State
----------------------------------------------------
 1     libvirt_test_api               running

# virsh vcpupin libvirt_test_api 0 2-8

# virsh vcpuinfo libvirt_test_api
VCPU:           0
CPU:            8
State:          running
CPU time:       8.6s
CPU Affinity:   --yyyyyyy-----------------------

VCPU:           1
CPU:            14
State:          running
CPU time:       2.0s
CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

VCPU:           2
CPU:            4
State:          running
CPU time:       2.4s
CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

VCPU:           3
CPU:            29
State:          running
CPU time:       2.2s
CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

VCPU:           4
CPU:            1
State:          running
CPU time:       2.9s
CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

VCPU:           5
CPU:            5
State:          running
CPU time:       2.2s
CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

VCPU:           6
CPU:            15
State:          running
CPU time:       2.1s
CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

VCPU:           7
CPU:            23
State:          running
CPU time:       1.9s
CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

# python
Python 2.6.6 (r266:84292, Aug 28 2012, 10:55:56) 
[GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import libvirt
>>> con = libvirt.open(None)
>>> dom = con.lookupByName('libvirt_test_api')
>>> dom.vcpus()
([(0, 1, 8570000000L, 8), (1, 1, 1990000000L, 14), (2, 1, 2370000000L, 4), (3, 1, 2160000000L, 29), (4, 1, 2960000000L, 8), (5, 1, 2240000000L, 5), (6, 1, 2150000000L, 15), (7, 1, 1940000000L, 23)], [(False, False, True, True, True, True, True, True, True, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False), (True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True), (True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True), (True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True), (True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True), (True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True), (True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True), (True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True)])
>>> dom.vcpuPinInfo(1)
[(False, False, True, True, True, True, True, True, True, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False), (True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True), (True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True), (True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True), (True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True), (True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True), (True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True), (True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True)]


also tried on machine with cpus as 4 and 64, it works.

Comment 7 hongming 2012-12-06 06:14:22 UTC
According to the result of Comment 6 .move its status to VERIFIED.

Comment 8 errata-xmlrpc 2013-02-21 07:26:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0276.html