Bug 1278174
Summary: | [RFE][nova]: Libvirt real time instances | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat OpenStack | Reporter: | Stephen Gordon <sgordon> | ||||
Component: | openstack-nova | Assignee: | Sahid Ferdjaoui <sferdjao> | ||||
Status: | CLOSED ERRATA | QA Contact: | Prasanth Anbalagan <panbalag> | ||||
Severity: | high | Docs Contact: | |||||
Priority: | high | ||||||
Version: | 8.0 (Liberty) | CC: | adahms, berrange, dasmith, eglynn, jschluet, kchamart, markmc, mlopes, ndipanov, nlevinki, panbalag, rhos-integ, sbauza, sferdjao, sgordon, smazziot, vromanso, yeylon | ||||
Target Milestone: | ga | Keywords: | FutureFeature, Reopened, TechPreview, TestOnly, Triaged, ZStream | ||||
Target Release: | 8.0 (Liberty) | ||||||
Hardware: | Unspecified | ||||||
OS: | Unspecified | ||||||
Whiteboard: | |||||||
Fixed In Version: | openstack-nova-12.0.1-4.el7ost | Doc Type: | Technology Preview | ||||
Doc Text: |
This package includes a technology preview of the real time feature for instances.
The real time feature provides stronger guarantees for 'worst-case' scheduler latency for vCPUs.
|
Story Points: | --- | ||||
Clone Of: | 1189551 | Environment: | |||||
Last Closed: | 2016-04-15 13:46:14 UTC | Type: | --- | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Bug Depends On: | 1189551, 1304381 | ||||||
Bug Blocks: | |||||||
Attachments: |
|
Comment 7
Stephen Gordon
2016-03-22 15:27:28 UTC
(In reply to Stephen Gordon from comment #7) > I believe you will however need the kernel-rt-kvm.x86_64 package and the RT > KVM module. Sorry typo, you need kernel-rt and kernel-rt-kvm. kernel-rt-kvm and the RT KVM module are of course the same thing. The doc text for this bug has been incorporated into the formal Technology Preview section, and does not need to be collected from this bug. Changing 'requires_doc_text' to '-'. Steve, I was able to test it on a deployment using packstack. But haven't been successful with a OSP-D deployment (not the feature, but just a 1 controller + 2 compute deployment). So I'm yet to test it on a deployment from OSP-D. Created attachment 1144922 [details]
Logs_failure_on_OSPD_deployment
1. Deploy RHOS-8 using steps detailed in OSP-setup-manual.txt (Check attachment logs.tar for the relevant .txt files) 2. Create image using cirros test image. glance image-create --container-format bare --disk-format qcow2 --file cirros-0.3.4-x86_64-disk.img --name cirros Note: You need to download cirros image from here http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img 3. Edit the following in nova.conf, a) vcpu_pin_set b) scheduler_default_filters 4. Create flavors as mentioned below and boot instances using the flavors. Booting instance with flavors 99 and 100 is successful, but flavor 101 will fail with error messages "Error launching a defined domain with XML: <domain type='kvm'>" and "libvirtError: Cannot set scheduler parameters for pid 28455: Operation not permitted" For detailed logs, please refer the libvirt and nova-compute logs in the attachment. nova flavor-create numa-topo-flv 99 512 5 2 nova flavor-key numa-topo-flv "hw:numa_nodes=2" "hw:numa_policy=preferred" nova flavor-key numa-topo-flv set "hw:numa_nodes=2" "hw:numa_policy=preferred" nova boot --flavor 99 --image cirros numa-test nova aggregate-create pinning nova aggregate-set-metadata 1 pinned=true nova host-list nova aggregate-add-host 1 overcloud-novacompute-0.localdomain nova aggregate-add-host 1 overcloud-novacompute-1.localdomain nova flavor-create m1.pinned 100 512 5 2 nova flavor-key 100 set hw:cpu_policy=dedicated nova flavor-key 100 set aggregate_instance_extra_specs:pinned=true nova boot --flavor 100 --image cirros vm1 nova flavor-create m1.realtime 101 512 5 2 nova flavor-key 101 set aggregate_instance_extra_specs:pinned=true nova flavor-key 101 set hw:cpu_policy=dedicated nova flavor-key 101 set hw:cpu_realtime=yes nova flavor-key 101 set hw:cpu_realtime_mask=^0 nova boot --flavor 101 --image cirros vm_realtime (In reply to Prasanth Anbalagan from comment #15) <spin> > 4. Create flavors as mentioned below and boot instances using the flavors. > Booting instance with flavors 99 and 100 is successful, but flavor 101 > will fail with error messages > "Error launching a defined domain with XML: <domain type='kvm'>" and > "libvirtError: Cannot set scheduler parameters for pid 28455: Operation > not permitted" > For detailed logs, please refer the libvirt and nova-compute logs in the > attachment. The error reported here is not in relation to Nova but probably host configuration, certainly you did not use good kernel RT. <snip> > nova flavor-create m1.realtime 101 512 5 2 > nova flavor-key 101 set aggregate_instance_extra_specs:pinned=true > nova flavor-key 101 set hw:cpu_policy=dedicated > nova flavor-key 101 set hw:cpu_realtime=yes > nova flavor-key 101 set hw:cpu_realtime_mask=^0 > nova boot --flavor 101 --image cirros vm_realtime (In reply to Sahid Ferdjaoui from comment #16) > (In reply to Prasanth Anbalagan from comment #15) > <spin> > > 4. Create flavors as mentioned below and boot instances using the flavors. > > Booting instance with flavors 99 and 100 is successful, but flavor 101 > > will fail with error messages > > "Error launching a defined domain with XML: <domain type='kvm'>" and > > "libvirtError: Cannot set scheduler parameters for pid 28455: Operation > > not permitted" > > For detailed logs, please refer the libvirt and nova-compute logs in the > > attachment. > > The error reported here is not in relation to Nova but probably host > configuration, certainly you did not use good kernel RT. According your logs it seems that you are using a kernel in an under version that the one I used. kernel-rt.x86_64 3.10.0-327.11.1.rt56.213.el7_2 kernel-rt-kvm.x86_64 3.10.0-327.11.1.rt56.213.el7_2 Can you try to update you kernel ? https://virt-wiki.lab.eng.brq.redhat.com/KVM/RealTime > <snip> > > nova flavor-create m1.realtime 101 512 5 2 > > nova flavor-key 101 set aggregate_instance_extra_specs:pinned=true > > nova flavor-key 101 set hw:cpu_policy=dedicated > > nova flavor-key 101 set hw:cpu_realtime=yes > > nova flavor-key 101 set hw:cpu_realtime_mask=^0 > > nova boot --flavor 101 --image cirros vm_realtime Sahid, Updating the kernel-rt and kernel-rt-kvm versions worked. [stack@instack ~]$ yum list installed | grep kernel-rt kernel-rt.x86_64 3.10.0-335.rt56.213.el7 kernel-rt-kvm.x86_64 3.10.0-335.rt56.213.el7 [heat-admin@overcloud-novacompute-1 ~]$ sudo virsh dumpxml 3 <domain type='kvm' id='3'> <name>instance-00000008</name> <uuid>6d4bd435-7040-49ba-92d5-7a1c6efddcc8</uuid> <metadata> <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0"> <nova:package version="12.0.2-5.el7ost"/> <nova:name>vm2</nova:name> <nova:creationTime>2016-04-11 14:52:12</nova:creationTime> <nova:flavor name="m1.realtime"> <nova:memory>512</nova:memory> <nova:disk>5</nova:disk> <nova:swap>0</nova:swap> <nova:ephemeral>0</nova:ephemeral> <nova:vcpus>2</nova:vcpus> </nova:flavor> <nova:owner> <nova:user uuid="edd5e36603fa40d79b119ef9afdf0681">admin</nova:user> <nova:project uuid="3de05fa407ae402ba9b9f68f5f0ff13e">admin</nova:project> </nova:owner> <nova:root type="image" uuid="33225a50-ffb0-4fb2-9341-794f7746fb5d"/> </nova:instance> </metadata> <memory unit='KiB'>524288</memory> <currentMemory unit='KiB'>524288</currentMemory> <memoryBacking> <nosharepages/> <locked/> </memoryBacking> <vcpu placement='static'>2</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='3'/> <emulatorpin cpuset='0'/> <vcpusched vcpus='1' scheduler='fifo' priority='1'/> </cputune> <numatune> <memory mode='strict' nodeset='0'/> <memnode cellid='0' mode='strict' nodeset='0'/> </numatune> <resource> <partition>/machine</partition> </resource> <sysinfo type='smbios'> <system> <entry name='manufacturer'>Red Hat</entry> <entry name='product'>OpenStack Compute</entry> <entry name='version'>12.0.2-5.el7ost</entry> <entry name='serial'>a0be2191-cd41-e246-a289-0bfbcad9c8e2</entry> <entry name='uuid'>6d4bd435-7040-49ba-92d5-7a1c6efddcc8</entry> <entry name='family'>Virtual Machine</entry> </system> </sysinfo> <os> <type arch='x86_64' machine='pc-i440fx-rhel7.2.0'>hvm</type> <boot dev='hd'/> <smbios mode='sysinfo'/> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-model'> <model fallback='allow'/> <topology sockets='1' cores='1' threads='2'/> <numa> <cell id='0' cpus='0-1' memory='524288' unit='KiB'/> </numa> </cpu> <clock offset='utc'> <timer name='pit' tickpolicy='delay'/> <timer name='rtc' tickpolicy='catchup'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/var/lib/nova/instances/6d4bd435-7040-49ba-92d5-7a1c6efddcc8/disk'/> <backingStore type='file' index='1'> <format type='raw'/> <source file='/var/lib/nova/instances/_base/f20886208a0ca20a5488ba8168b583f36da5ee20'/> <backingStore/> </backingStore> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </disk> <controller type='usb' index='0'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <serial type='tcp'> <source mode='bind' host='127.0.0.1' service='10001'/> <protocol type='raw'/> <target port='0'/> <alias name='serial0'/> </serial> <serial type='pty'> <source path='/dev/pts/2'/> <target port='1'/> <alias name='serial1'/> </serial> <console type='tcp'> <source mode='bind' host='127.0.0.1' service='10001'/> <protocol type='raw'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <input type='tablet' bus='usb'> <alias name='input0'/> </input> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='5901' autoport='yes' listen='0.0.0.0' keymap='en-us'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='cirrus' vram='16384' heads='1'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <stats period='10'/> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='selinux' relabel='yes'> <label>system_u:system_r:svirt_t:s0:c84,c914</label> <imagelabel>system_u:object_r:svirt_image_t:s0:c84,c914</imagelabel> </seclabel> </domain> If the kernel version shipped with RHOS 8 is not good a bug should have been reported, did you report one ? Sahid, Here is the bug information. https://bugzilla.redhat.com/show_bug.cgi?id=1326154 Verified that kernel-rt and kernel-rt-kvm versions 3.10.0-327.13.1.rt56.216.el7_2 also works for this RFE. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0636.html |