This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1262069 - [RFE] Dynamically apply an IOPS limit based on a ceilometer trigger to limit a "noisy" VM
[RFE] Dynamically apply an IOPS limit based on a ceilometer trigger to limit ...
Status: NEW
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova (Show other bugs)
9.0 (Mitaka)
Unspecified Unspecified
low Severity low
: ---
: ---
Assigned To: Eoghan Glynn
nlevinki
: FutureFeature
Depends On: 1369482 1261092
Blocks:
  Show dependency treegraph
 
Reported: 2015-09-10 14:39 EDT by Neil Levine
Modified: 2017-07-28 13:06 EDT (History)
13 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Neil Levine 2015-09-10 14:39:21 EDT
If we see a VM consuming more IOPS than allowed by a threshold set in Ceilomater, we want to automatically limit it so it doesn't affect other tenants. This will provide a basic QoS facility.
Comment 3 Stephen Gordon 2016-01-29 09:54:35 EST
(In reply to Neil Levine from comment #0)
> If we see a VM consuming more IOPS than allowed by a threshold set in
> Ceilomater, we want to automatically limit it so it doesn't affect other
> tenants. This will provide a basic QoS facility.

Can you elaborate on what is needed over and above instance resource quotas:

    https://wiki.openstack.org/wiki/InstanceResourceQuota

Thanks,

Steve
Comment 4 Neil Levine 2016-01-29 14:01:48 EST
Those settings are the ones we need buto I think this is actually a qemu-kvm issue not a nova one, i.e qemu-kvm doesn't honor the IOPs limits when applied to RBD volumes. 

Josh, is that correct?

If so, we need to reassign the ticket to the qemu-kvm component.
Comment 5 Josh Durgin 2016-01-29 14:19:22 EST
qemu does honor I/O limits for rbd, using its own throttling. cgroups limits do not work with librbd of course. There's a bug in previous nova versions that would not pass the disk io limits to qemu for rbd.

What I'm not sure about is whether these limits can be changed after a device is attached. Nova doesn't support that, and I'm not sure whether qemu does.
Comment 6 Neil Levine 2016-01-29 14:28:17 EST
So is this a bug BZ and not a RFE against nova?
Comment 7 Josh Durgin 2016-01-29 14:35:07 EST
If you don't mind statically defining the limits, it's a bug. Specifically this one:

https://bugs.launchpad.net/nova/+bug/1405367

If you wanted to adjust the limits dynamically, it'd be a nova RFE and possibly also a qemu RFE.
Comment 8 Neil Levine 2016-01-29 14:40:10 EST
We need the dynamic limits as the user story around this feature is to help admins clamp on noisy VMs. 

Just having the static limits work for now would be a win though.
Comment 9 Stephen Gordon 2016-01-29 17:27:33 EST
(In reply to Josh Durgin from comment #7)
> If you don't mind statically defining the limits, it's a bug. Specifically
> this one:
> 
> https://bugs.launchpad.net/nova/+bug/1405367

OK, we were actually already tracking this under Bug # 1261092 which has been verified for RHEL OpenStack Platform 8.

> If you wanted to adjust the limits dynamically, it'd be a nova RFE and
> possibly also a qemu RFE.

This is more complicated, since as a general comment Nova instances are currently "fire and forget" and there isn't really an interface to change things after the fact.

There is a proposal up around live resize which is more specifically targeted at adding CPU/RAM/Disk to a running instance but I am wondering if this type of change would fit as an extension or that or be shouted down.
Comment 10 Stephen Gordon 2016-01-29 17:30:30 EST
One other comment, the full use case - "apply an IOPS limit based on a ceilometer trigger to limit a "noisy" VM" - sounds like it would belong being orchestrated by a higher level system (e.g. CloudForms) I think what we need in Ceilometer/Nova is to ensure we are exposing the right information and APIs for such a system to use though.
Comment 11 Neil Levine 2016-03-17 19:25:09 EDT
(In reply to Stephen Gordon from comment #10)
> One other comment, the full use case - "apply an IOPS limit based on a
> ceilometer trigger to limit a "noisy" VM" - sounds like it would belong
> being orchestrated by a higher level system (e.g. CloudForms) I think what
> we need in Ceilometer/Nova is to ensure we are exposing the right
> information and APIs for such a system to use though.

Yeah, this makes sense. We're starting to look at native QoS in Ceph which will address the higher-level use-case but getting VM-centric logging of IOPS is always going to be necessary.

Is there info on what Nova (or Qemu) logs for VM IOPS currently?

Note You need to log in before you can comment on or make changes to this bug.