Bug 1006381 - nova [Horizon]: all instances run is failing when exceeding quota on multiple instance launch
Summary: nova [Horizon]: all instances run is failing when exceeding quota on multiple...
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: RDO
Classification: Community
Component: python-django-horizon
Version: unspecified
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
: ---
Assignee: Matthias Runge
QA Contact: Ami Jeain
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-09-10 14:18 UTC by Dafna Ron
Modified: 2015-06-04 21:53 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-06-18 07:11:42 UTC


Attachments (Terms of Use)
log (2.95 KB, application/x-xz)
2013-09-10 14:18 UTC, Dafna Ron
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1331341 0 None None None Never

Description Dafna Ron 2013-09-10 14:18:42 UTC
Created attachment 796024 [details]
log

Description of problem:

I ran a 100 instances with largest flavour on AIO 
all instances fail to run on quota. 
 
Version-Release number of selected component (if applicable):

openstack-nova-compute-2013.1.3-3.el6ost.noarch

How reproducible:

100%

Steps to Reproduce:
1. install AIO
2. launch 100 instances with biggest flavour 
3.

Actual results:

quota fails the run of all instances 

Expected results:
we should run what we can and fail the rest 

Additional info: api log

Comment 1 Dafna Ron 2013-09-10 14:49:14 UTC
https://bugs.launchpad.net/nova/+bug/1223374

Comment 2 Xavier Queralt 2013-09-11 11:04:40 UTC
Actually nova does what you ask. It will launch the instances that fit within the current quota limits. On the other hand, the dashboard will prevent you from doing this.

See below an example where I try to boot two instances with flavor m1.nano (128MB of ram) while the user can't use more than 200MB of ram. The call won't fail but nova will only boot one of the two requested instances.

[root@rdo1 ~(keystone_admin)]# rpm -qa openstack-nova-compute
openstack-nova-compute-2013.2-0.19.b3.fc20.noarch

[root@rdo1 ~(keystone_admin)]# nova flavor-show m1.nano
+----------------------------+--------------------------------------+
| Property                   | Value                                |
+----------------------------+--------------------------------------+
| name                       | m1.nano                              |
| ram                        | 128                                  |
| OS-FLV-DISABLED:disabled   | False                                |
| vcpus                      | 1                                    |
| extra_specs                | {}                                   |
| swap                       |                                      |
| os-flavor-access:is_public | True                                 |
| rxtx_factor                | 1.0                                  |
| OS-FLV-EXT-DATA:ephemeral  | 0                                    |
| disk                       | 1                                    |
| id                         | dbfca59b-cf4e-4c10-9342-e1c49145c1ff |
+----------------------------+--------------------------------------+

[root@rdo1 ~(keystone_admin)]# nova quota-show
+-----------------------------+-------+
| Property                    | Value |
+-----------------------------+-------+
| metadata_items              | 128   |
| injected_file_content_bytes | 10240 |
| ram                         | 200   |
| floating_ips                | 10    |
| key_pairs                   | 100   |
| instances                   | 10    |
| security_group_rules        | 20    |
| injected_files              | 5     |
| cores                       | 20    |
| fixed_ips                   | -1    |
| injected_file_path_bytes    | 255   |
| security_groups             | 10    |
+-----------------------------+-------+

[root@rdo1 ~(keystone_admin)]# nova boot --image cirros --flavor m1.nano --num-instances 2 nuvol

[root@rdo1 fedora(keystone_admin)]# nova list
+--------------------------------------+-------+--------+------------+-------------+--------------------------+
| ID                                   | Name  | Status | Task State | Power State | Networks                 |
+--------------------------------------+-------+--------+------------+-------------+--------------------------+
| c658fd31-687e-4a4e-b05e-7da408e09f0a | nuvol | BUILD  | spawning   | NOSTATE     | novanetwork=192.168.32.3 |
+--------------------------------------+-------+--------+------------+-------------+--------------------------+

Comment 3 Dafna Ron 2013-09-11 12:34:06 UTC
so should this be handled by horizon?

Comment 4 Xavier Queralt 2013-09-19 09:22:02 UTC
If this has to be handled should be on horizon side.

Although I don't see any problem with the current behaviour. The dashboard will show you graphically how much resources you have available (taken from your quota) when launching a new instance. Why would you want to boot a bigger number of instances if you see they don't fit?

I'm moving this to horizon component.

Comment 5 Alvaro Lopez Ortega 2014-01-08 09:56:52 UTC
Liz, could you please give us a hand here? What'd be the right approach?

Comment 6 Liz 2014-01-08 15:40:09 UTC
Hi All,

It does sounds like this is something that we need to approach as an improvement to the Horizon UI.

Currently, the user does get an error if they attempt to instances that go over their quota in some way but it isn't presented until after they select the "Launch" button in the Launch Instance modal. The error they are given is a pop-up letting them know that the system couldn't add any of the instances:
http://file.bos.redhat.com/~lsurette/OS%20Community/Add%20Instance%20Error.png

Now the user has to go back into the modal and fill out all of the information again about what they'd like to launch, making sure they fix the error from before.

I think that this should perform consistently with when a user fills out a field in the wrong way and give the user an error on the "Number of Instances" field with a message explaining that this action can't be performed due to the quota limitations. This way the user can quickly update the number of instances or even they could update the size of the instance if they want to get certain quota numbers down. Here is an example of an error on one of the fields in this modal that I would expect this error to act like:
http://file.bos.redhat.com/~lsurette/OS%20Community/Error%20within%20Launch%20Instance%20Modal.png

I think the message would be best if it read out something like "VCPU quota exceeded."


My thoughts,
Liz

Comment 7 Matthias Runge 2014-01-23 13:34:17 UTC
Liz, that would work, when there is only one user per tenant. 

But you can have changes between showing the form and submitting, i.e another user for the tenant launched a significant number of instances.

So the question, is still: what to do, if a launch request to launch a number of instances fails. There are two possibilities:

- launch as many as possible instances, notifying the user about successes and failures
- do not launch a single instance. This makes sense, if the user really requires that number of instances.

Comment 8 Liz 2014-01-27 19:19:53 UTC
Ah, understood that there could be others adding instances while I have the modal up.

Could we do another check on the click of "Launch" in the modal and then if there isn't space, bring the modal back up with an error?

If not, I'd vote for launching as many instances as we can and making the error message very clear, something like "Only 5 out of the 8 requested instances could be launched due to quota limitations."

Liz

Comment 9 Matthias Runge 2014-06-18 07:11:42 UTC
reported upstream, https://bugs.launchpad.net/horizon/+bug/1331341

for reference, the launch wizard prevents you from launching more instances than your quota allows you to do.


Note You need to log in before you can comment on or make changes to this bug.