Bug 1609785

* Description of problem:
  Starting <cpu ... <numa> guest domains on systems with unavailable cpu resources causes libvirt to get into an unworkable state reporting

  error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d3\x2dvnuma.scope/vcpu0/cpuset.cpus': Permission denied

  It seems from not possible regain functionality by adding the missing cpu resources and restart libvirt.

* Version-Release number of selected component (if applicable):
  Latest Fedora (from F26, F27, F28, ... CentOS, RHT?)

* How reproducible:
  100%

* Steps to Reproduce:
  1. Administrate a KVM guest domain based on NUMA h/w with below pinning, (specific) numa topology added. This is a 4-node NUMA setup with 120 cpus.  See the vcpupin detail below to understand its topology.  We simply setup a mini NUMA architecture for 5 vcpus where 0,4 are set to hold affinity with NODE0, 1 with NODE1, 2 with NODE2 and 3 with NODE3.

     <domain type='kvm'>
       <name>vnuma</name>
         ...
       <vcpu placement='static' current='5'>8</vcpu>
       <cputune>
         <vcpupin vcpu='0' cpuset='0-14,60-74'/>
         <vcpupin vcpu='1' cpuset='15-29,75-89'/>
         <vcpupin vcpu='2' cpuset='30-44,90-104'/>
         <vcpupin vcpu='3' cpuset='45-59,105-119'/>
         <vcpupin vcpu='4' cpuset='0-14,60-74'/>
       </cputune>
         ...
      <cpu mode='host-model' check='none'>
        <model fallback='allow'/>
        <topology sockets='4' cores='1' threads='2'/>
        <numa>
          <cell id='0' cpus='0,4' memory='2097152' unit='KiB'>
            <distances>
              <sibling id='0' value='10'/>
              <sibling id='1' value='21'/>
              <sibling id='2' value='31'/>
              <sibling id='3' value='21'/>
            </distances>
          </cell>
          <cell id='1' cpus='1' memory='2097152' unit='KiB'>
            <distances>
              <sibling id='0' value='21'/>
              <sibling id='1' value='10'/>
              <sibling id='2' value='21'/>
              <sibling id='3' value='31'/>
            </distances>
          </cell>
          <cell id='2' cpus='2' memory='2097152' unit='KiB'>
          <distances>
            <sibling id='0' value='31'/>
            <sibling id='1' value='21'/>
            <sibling id='2' value='10'/>
            <sibling id='3' value='21'/>
          </distances>
        </cell>
        <cell id='3' cpus='3' memory='2097152' unit='KiB'>
          <distances>
            <sibling id='0' value='21'/>
            <sibling id='1' value='31'/>
            <sibling id='2' value='21'/>
            <sibling id='3' value='10'/>
          </distances>
        </cell>
      </numa>
    </cpu>
      ..
  </domain>

  Before going to step 2. test if the above works.  It should.  You should be able to create a guest domain no based on 4 NUMA administrated cells.  So, simply said (below should work prior continuing);
  sudo virsh start vnuma
  sudo virsh shutdown vnuma

  2. Have the 'vnuma' guest domain shutdown.
     Take all physical node NUMA cpu's offline apart from those running under NODE0.

     # lscpu 
     Architecture:          x86_64
     CPU op-mode(s):        32-bit, 64-bit
     Byte Order:            Little Endian
     CPU(s):                120
     On-line CPU(s) list:   0-119
     Thread(s) per core:    2
     Core(s) per socket:    15
     Socket(s):             4
     NUMA node(s):          4
     Vendor ID:             GenuineIntel
     CPU family:            6
     Model:                 62
     Model name:            Intel(R) Xeon(R) CPU E7-8895 v2 @ 2.80GHz
     Stepping:              7
     CPU MHz:               3529.736
     CPU max MHz:           3600.0000
     CPU min MHz:           1200.0000
     BogoMIPS:              5586.77
     Virtualization:        VT-x
     L1d cache:             32K
     L1i cache:             32K
     L2 cache:              256K
     L3 cache:              38400K
     NUMA node0 CPU(s):     0-14,60-74
     NUMA node1 CPU(s):     15-29,75-89
     NUMA node2 CPU(s):     30-44,90-104
     NUMA node3 CPU(s):     45-59,105-119

     # chcpu -d 15-119
     CPU 15 disabled
     CPU 16 disabled
     CPU 17 disabled
     CPU 18 disabled
       ...
     CPU 119 disabled

  
  3. Try to start the 'vnuma' guest domain again (see 1.)
     virsh # start --console vnuma 
     error: Failed to start domain vnuma
     error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d3\x2dvnuma.scope/vcpu0/cpuset.cpus': Permission denied

  4. Try to reonline the physical CPUs.  Try to restart libvirt ... find that your last resort is to restart the entire machine (BAD) :-(


* Actual results:
  Broken ... not highly available.  Machine needs a reboot to recover.   

* Expected results:
  Expected not to see this bug.  Eventually expected to have seen a reasonable error messages.  Also expected to see a possibility to recover without rebooting the machine.


* Additional info:
  none.