Description of problem:
The customer is observing high CPU steal values on instances on specific (most of) his hypervisors. After analysis of all hypervisors, it seems that most vCPUs get mostly scheduled on CPU 0 of the hypervisors. These hypervisors were configured with isolcpus kernel command line parameter. The customer is aware that this is a bad configuration and that they will have to remove this eventually.

Theory:
- some bug in the scheduler (possibly triggered due to isolcpus) puts most vCPUs on CPU 0 and thus creates high contention for that CPU and high steal values within the VMs

Version-Release number of selected component (if applicable):
libvirt-client-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-interface-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-network-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-nodedev-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-nwfilter-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-qemu-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-secret-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-storage-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-kvm-1.2.17-13.el7_2.4.x86_64
libvirt-python-1.2.17-2.el7.x86_64


How reproducible:
The issue can be observed on most hypervisors which we did not take action upon yet:
- run ` virsh list | awk '{print $2}' | xargs -I {} virsh vcpuinfo {} | egrep '^CPU\:' | awk '{print $NF}' | sort | uniq -c | sort -nr` on all hypervisors to find the top scheduled CPUs at a given moment
~~~
d4-ucos-nova4 | SUCCESS | rc=0 >>
     39 0
      9 8
      9 6
(...)
d4-ucos-nova7 | SUCCESS | rc=0 >>
     55 0
      4 21
      4 13
      3 22
      3 18
(...)
d4-ucos-nova8 | SUCCESS | rc=0 >>
     44 0
      5 9
      4 5
      4 4
(...)
d4-ucos-nova10 | SUCCESS | rc=0 >>
     43 0
      7 9
      6 8
      6 7
      6 5
(...)
d4-ucos-nova14 | SUCCESS | rc=0 >>
     21 0
      3 21
      2 9
      2 7
(...)
~~~

The ask:
We do have a mitigation for this scheduling issue, but need an explanation and permanent fix.

How to mitigate the issue:
- create script `pinning.sh`. Modify reserve_first to change the CPU Affinity of all VMs on a given hypervisor to `${reserve_first}-$[ $cpu_count -1 ]`
~~~
#!/bin/bash
#
#################################################
# This script takes an instance name, pins all of its vCPUs to the first hypervisor vcpus and lets all other VMs' vcpus roam freely among the rest
# 2016 - Red Hat - akaris
#################################################

cpu_count=`lscpu | egrep '^CPU\(s\)' | awk '{print $NF}'`
reserve_first=0

echo "Adjusting pinning for all instances"
virsh list | awk '{print $2}' | tail -n+3 | head -n-1 | while read instance;do
  if [ "$instance" == "$reserve_instance" ];then
    echo "Skipping instance $instance"
    continue
  fi
  echo "Adjusting pinning for $instance"
  virsh vcpupin $instance | egrep '^\s+[0-9]' | awk -F ':' '{print $1}' | while read vcpu;do
    virsh vcpupin $instance $vcpu ${reserve_first}-$[ $cpu_count -1 ]
  done
  virsh vcpupin $instance
done

echo ""
echo "==============================================="
echo "Verification output"
echo "==============================================="
virsh list | awk '{print $2}' | xargs -I {} bash -c "echo {}; virsh vcpupin {}" 2>/dev/null
~~~
- run pinning.sh with `reserve_first=5` (5 was chosen randomly. We would need to investigate if 4 is the first effective value, if so, then this may likely have something to do with `isolcpu`)
- run pinning.sh with `reserve_first=0`
- observe that CPU 0 is not being scheduled any more

Example on nova2 for mitigation procedure:
same issue on nova2
~~~
[root@d4-ucos-nova2 ~]# ./vcpu_scheduling.sh
error: failed to get domain 'Name'
error: Domain not found: no domain with matching name 'Name'
     78 0
     11 22
      7 5
      7 20
      5 23
      4 9
      4 8
      3 6
      3 4
      3 19
      3 15
      3 14
      2 7
      2 21
      2 18
      2 13
      2 10
      1 28
      1 27
      1 12
      1 11
~~~

~~~
[root@d4-ucos-nova2 ~]#  virsh list | awk '{print $2}' | xargs -I {} virsh vcpuinfo {} | grep Aff  | uniq -c
error: failed to get domain 'Name'
error: Domain not found: no domain with matching name 'Name'
    145 CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
~~~

ran pinning.sh with 0-39 .. did not change anything ...

~~~
[root@d4-ucos-nova2 ~]#  ./vcpu_scheduling.sh
error: failed to get domain 'Name'
error: Domain not found: no domain with matching name 'Name'
     79 0
      9 7
      8 23
      7 22
      5 21
      4 6
      4 5
      3 9
      3 8
      3 4
      3 18
      2 20
      2 19
      2 16
      2 13
      2 12
      1 39
      1 38
      1 27
      1 17
      1 14
      1 11
      1 10
~~~

changing 1-39
~~~
[root@d4-ucos-nova2 ~]# ./vcpu_scheduling.sh
error: failed to get domain 'Name'
error: Domain not found: no domain with matching name 'Name'
     79 1
     10 23
      6 6
      5 4
      5 19
      5 14
      4 9
      4 22
      4 21
      4 20
      3 8
      3 7
      3 5
      3 17
      2 15
      2 10
      1 18
      1 16
      1 11
~~~

changing 5-39:
~~~
instance-00007656
VCPU: CPU Affinity
----------------------------------
   0: 5-39
   1: 5-39

[root@d4-ucos-nova2 ~]# ./vcpu_scheduling.sh
error: failed to get domain 'Name'
error: Domain not found: no domain with matching name 'Name'
     20 23
     18 7
     17 22
     15 5
     14 8
     11 9
     11 21
      7 6
      7 24
      7 20
      4 17
      2 33
      2 28
      2 18
      2 16
      1 27
      1 26
      1 19
      1 13
      1 12
      1 10
~~~

changing back to 0-39
~~~
instance-00007656
VCPU: CPU Affinity
----------------------------------
   0: 0-39
   1: 0-39

[root@d4-ucos-nova2 ~]# ./vcpu_scheduling.sh
error: failed to get domain 'Name'
error: Domain not found: no domain with matching name 'Name'
     19 9
     17 23
     17 22
     14 5
     10 21
      9 8
      9 6
      8 7
      7 20
      4 11
      3 4
      3 27
      3 25
      3 19
      3 17
      3 16
      3 15
      2 28
      2 13
      1 35
      1 33
      1 29
      1 24
      1 18
      1 10
~~~