Bug 1278174 - [RFE][nova]: Libvirt real time instances
[RFE][nova]: Libvirt real time instances
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova (Show other bugs)
8.0 (Liberty)
Unspecified Unspecified
high Severity high
: ga
: 8.0 (Liberty)
Assigned To: Sahid Ferdjaoui
Prasanth Anbalagan
: FutureFeature, Reopened, TechPreview, TestOnly, Triaged, ZStream
Depends On: 1189551 1304381
Blocks:
  Show dependency treegraph
 
Reported: 2015-11-04 15:39 EST by Stephen Gordon
Modified: 2016-04-15 09:46 EDT (History)
19 users (show)

See Also:
Fixed In Version: openstack-nova-12.0.1-4.el7ost
Doc Type: Technology Preview
Doc Text:
This package includes a technology preview of the real time feature for instances. The real time feature provides stronger guarantees for 'worst-case' scheduler latency for vCPUs.
Story Points: ---
Clone Of: 1189551
Environment:
Last Closed: 2016-04-15 09:46:14 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Logs_failure_on_OSPD_deployment (314.64 KB, application/x-gzip)
2016-04-07 19:28 EDT, Prasanth Anbalagan
no flags Details

  None (edit)
Comment 7 Stephen Gordon 2016-03-22 11:27:28 EDT
I believe you will however need the kernel-rt-kvm.x86_64 package and the RT KVM module.
Comment 8 Stephen Gordon 2016-03-22 11:28:10 EDT
(In reply to Stephen Gordon from comment #7)
> I believe you will however need the kernel-rt-kvm.x86_64 package and the RT
> KVM module.

Sorry typo, you need kernel-rt and kernel-rt-kvm. kernel-rt-kvm and the RT KVM module are of course the same thing.
Comment 9 Andrew Dahms 2016-03-30 21:19:44 EDT
The doc text for this bug has been incorporated into the formal Technology Preview section, and does not need to be collected from this bug.

Changing 'requires_doc_text' to '-'.
Comment 11 Prasanth Anbalagan 2016-04-04 20:35:08 EDT
Steve,

I was able to test it on a deployment using packstack. But haven't been successful with a OSP-D deployment (not the feature, but just a 1 controller + 2 compute deployment). So I'm yet to test it on a deployment from OSP-D.
Comment 14 Prasanth Anbalagan 2016-04-07 19:28 EDT
Created attachment 1144922 [details]
Logs_failure_on_OSPD_deployment
Comment 15 Prasanth Anbalagan 2016-04-07 19:30:53 EDT
1. Deploy RHOS-8 using steps detailed in OSP-setup-manual.txt
    (Check attachment logs.tar for the relevant .txt files)

2. Create image using cirros test image.
     glance image-create --container-format bare --disk-format qcow2 --file cirros-0.3.4-x86_64-disk.img --name cirros
     Note: You need to download cirros image from here
      http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img
      
3. Edit the following in nova.conf,
    a) vcpu_pin_set
    b) scheduler_default_filters
    
4. Create flavors as mentioned below and boot instances using the flavors.
   Booting instance with flavors 99 and 100 is successful, but flavor 101 
   will fail with error messages 
   "Error launching a defined domain with XML: <domain type='kvm'>" and 
   "libvirtError: Cannot set scheduler parameters for pid 28455: Operation not permitted"
   For detailed logs, please refer the libvirt and nova-compute logs in the attachment.
   
   nova flavor-create numa-topo-flv 99 512 5 2
   nova flavor-key numa-topo-flv "hw:numa_nodes=2" "hw:numa_policy=preferred"
   nova flavor-key numa-topo-flv set "hw:numa_nodes=2" "hw:numa_policy=preferred"
   nova boot --flavor 99 --image cirros numa-test
   
   
   nova aggregate-create pinning
   nova aggregate-set-metadata 1 pinned=true
   nova host-list
   nova aggregate-add-host 1 overcloud-novacompute-0.localdomain
   nova aggregate-add-host 1 overcloud-novacompute-1.localdomain
   nova flavor-create m1.pinned 100 512 5 2
   nova flavor-key 100 set hw:cpu_policy=dedicated
   nova flavor-key 100 set aggregate_instance_extra_specs:pinned=true
   nova boot --flavor 100 --image cirros vm1
   
   
   nova flavor-create m1.realtime 101 512 5 2
   nova flavor-key 101 set aggregate_instance_extra_specs:pinned=true
   nova flavor-key 101 set hw:cpu_policy=dedicated
   nova flavor-key 101 set hw:cpu_realtime=yes
   nova flavor-key 101 set hw:cpu_realtime_mask=^0
   nova boot --flavor 101 --image cirros vm_realtime
Comment 16 Sahid Ferdjaoui 2016-04-11 06:47:30 EDT
(In reply to Prasanth Anbalagan from comment #15)
<spin>
> 4. Create flavors as mentioned below and boot instances using the flavors.
>    Booting instance with flavors 99 and 100 is successful, but flavor 101 
>    will fail with error messages 
>    "Error launching a defined domain with XML: <domain type='kvm'>" and 
>    "libvirtError: Cannot set scheduler parameters for pid 28455: Operation
> not permitted"
>    For detailed logs, please refer the libvirt and nova-compute logs in the
> attachment.

The error reported here is not in relation to Nova but probably host configuration, certainly you did not use good kernel RT.

<snip>    
>    nova flavor-create m1.realtime 101 512 5 2
>    nova flavor-key 101 set aggregate_instance_extra_specs:pinned=true
>    nova flavor-key 101 set hw:cpu_policy=dedicated
>    nova flavor-key 101 set hw:cpu_realtime=yes
>    nova flavor-key 101 set hw:cpu_realtime_mask=^0
>    nova boot --flavor 101 --image cirros vm_realtime
Comment 17 Sahid Ferdjaoui 2016-04-11 07:02:57 EDT
(In reply to Sahid Ferdjaoui from comment #16)
> (In reply to Prasanth Anbalagan from comment #15)
> <spin>
> > 4. Create flavors as mentioned below and boot instances using the flavors.
> >    Booting instance with flavors 99 and 100 is successful, but flavor 101 
> >    will fail with error messages 
> >    "Error launching a defined domain with XML: <domain type='kvm'>" and 
> >    "libvirtError: Cannot set scheduler parameters for pid 28455: Operation
> > not permitted"
> >    For detailed logs, please refer the libvirt and nova-compute logs in the
> > attachment.
> 
> The error reported here is not in relation to Nova but probably host
> configuration, certainly you did not use good kernel RT.

According your logs it seems that you are using a kernel in an under version that the one I used.

kernel-rt.x86_64                     3.10.0-327.11.1.rt56.213.el7_2
kernel-rt-kvm.x86_64                 3.10.0-327.11.1.rt56.213.el7_2

Can you try to update you kernel ? https://virt-wiki.lab.eng.brq.redhat.com/KVM/RealTime


> <snip>    
> >    nova flavor-create m1.realtime 101 512 5 2
> >    nova flavor-key 101 set aggregate_instance_extra_specs:pinned=true
> >    nova flavor-key 101 set hw:cpu_policy=dedicated
> >    nova flavor-key 101 set hw:cpu_realtime=yes
> >    nova flavor-key 101 set hw:cpu_realtime_mask=^0
> >    nova boot --flavor 101 --image cirros vm_realtime
Comment 18 Prasanth Anbalagan 2016-04-11 10:59:00 EDT
Sahid,

Updating the kernel-rt and kernel-rt-kvm versions worked. 

[stack@instack ~]$ yum list installed | grep kernel-rt
kernel-rt.x86_64                 3.10.0-335.rt56.213.el7
kernel-rt-kvm.x86_64             3.10.0-335.rt56.213.el7

[heat-admin@overcloud-novacompute-1 ~]$ sudo virsh dumpxml 3
<domain type='kvm' id='3'>
  <name>instance-00000008</name>
  <uuid>6d4bd435-7040-49ba-92d5-7a1c6efddcc8</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="12.0.2-5.el7ost"/>
      <nova:name>vm2</nova:name>
      <nova:creationTime>2016-04-11 14:52:12</nova:creationTime>
      <nova:flavor name="m1.realtime">
        <nova:memory>512</nova:memory>
        <nova:disk>5</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>2</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="edd5e36603fa40d79b119ef9afdf0681">admin</nova:user>
        <nova:project uuid="3de05fa407ae402ba9b9f68f5f0ff13e">admin</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="33225a50-ffb0-4fb2-9341-794f7746fb5d"/>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>524288</memory>
  <currentMemory unit='KiB'>524288</currentMemory>
  <memoryBacking>
    <nosharepages/>
    <locked/>
  </memoryBacking>
  <vcpu placement='static'>2</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='3'/>
    <emulatorpin cpuset='0'/>
    <vcpusched vcpus='1' scheduler='fifo' priority='1'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
    <memnode cellid='0' mode='strict' nodeset='0'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>Red Hat</entry>
      <entry name='product'>OpenStack Compute</entry>
      <entry name='version'>12.0.2-5.el7ost</entry>
      <entry name='serial'>a0be2191-cd41-e246-a289-0bfbcad9c8e2</entry>
      <entry name='uuid'>6d4bd435-7040-49ba-92d5-7a1c6efddcc8</entry>
      <entry name='family'>Virtual Machine</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.2.0'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-model'>
    <model fallback='allow'/>
    <topology sockets='1' cores='1' threads='2'/>
    <numa>
      <cell id='0' cpus='0-1' memory='524288' unit='KiB'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/nova/instances/6d4bd435-7040-49ba-92d5-7a1c6efddcc8/disk'/>
      <backingStore type='file' index='1'>
        <format type='raw'/>
        <source file='/var/lib/nova/instances/_base/f20886208a0ca20a5488ba8168b583f36da5ee20'/>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </disk>
    <controller type='usb' index='0'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <serial type='tcp'>
      <source mode='bind' host='127.0.0.1' service='10001'/>
      <protocol type='raw'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <serial type='pty'>
      <source path='/dev/pts/2'/>
      <target port='1'/>
      <alias name='serial1'/>
    </serial>
    <console type='tcp'>
      <source mode='bind' host='127.0.0.1' service='10001'/>
      <protocol type='raw'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='5901' autoport='yes' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <stats period='10'/>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c84,c914</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c84,c914</imagelabel>
  </seclabel>
</domain>
Comment 19 Sahid Ferdjaoui 2016-04-11 11:09:55 EDT
If the kernel version shipped with RHOS 8 is not good a bug should have been reported, did you report one ?
Comment 20 Prasanth Anbalagan 2016-04-11 22:04:13 EDT
Sahid,

Here is the bug information. 

https://bugzilla.redhat.com/show_bug.cgi?id=1326154
Comment 21 Prasanth Anbalagan 2016-04-12 14:55:21 EDT
Verified that kernel-rt and kernel-rt-kvm versions 3.10.0-327.13.1.rt56.216.el7_2
also works for this RFE.
Comment 23 errata-xmlrpc 2016-04-15 09:46:14 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0636.html

Note You need to log in before you can comment on or make changes to this bug.