Bug 1376441

Error received on horizon:

"The requested instance cannot be launched. The following requested resource(s) exceed quota(s): Cores(Available: 0, Requested: 1)."

From the code below:


dashboards/project/instances/workflows/create_instance.py



    def _check_quotas(self, cleaned_data):
        count = cleaned_data.get('count', 1)

        # Prevent launching more instances than the quota allows
        usages = quotas.tenant_quota_usages(self.request)
        available_count = usages['instances']['available']
        if available_count < count:
            error_message = ungettext_lazy(
                'The requested instance cannot be launched as you only '
                'have %(avail)i of your quota available. ',
                'The requested %(req)i instances cannot be launched as you '
                'only have %(avail)i of your quota available.',
                count)
            params = {'req': count,
                      'avail': available_count}
            raise forms.ValidationError(error_message % params)

        flavor_id = cleaned_data.get('flavor')
        flavor = self._get_flavor(flavor_id)  

        count_error = []
        # Validate cores and ram.
        available_cores = usages['cores']['available']
        if flavor and available_cores < count * flavor.vcpus:
            count_error.append(_("Cores(Available: %(avail)s, "
                                 "Requested: %(req)s)")
                               % {'avail': available_cores,
                                  'req': count * flavor.vcpus})

        available_ram = usages['ram']['available']
        if flavor and available_ram < count * flavor.ram:
            count_error.append(_("RAM(Available: %(avail)s, "
                                 "Requested: %(req)s)")
                               % {'avail': available_ram,
                                  'req': count * flavor.ram})

        if count_error:
            value_str = ", ".join(count_error)
            msg = (_('The requested instance cannot be launched. '
                     'The following requested resource(s) exceed '
                     'quota(s): %s.') % value_str)
            if count == 1:
                self._errors['flavor'] = self.error_class([msg])
            else:
                self._errors['count'] = self.error_class([msg])



We're getting available cores as 0, this is gathered from "usages['cores']['available']"

Created attachment 1212811 [details]
the quota_usages table for that tenant

Created attachment 1212812 [details]
the live rows from the instances table for that tenant

Created attachment 1216527 [details]
nova flavor-show output

Hello,

please find attached the requested information.


Seb.

Hi,

I still didn't manage to pinpoint the source of discrepancy between the Nova's and Horizon's calculations, other than Nova reports vcpus numbers from actually deployed instances, while Horizon uses the numbers defined in their flavors. Those numbers should be the same, unless the flavors have been tampered with since the instances were deployed -- but I assume that we would have been told about it if that was the case.

I will try to ask the Nova team for advice again.

We will need some additional data. Can you please ask the customer for dumps of the tables "flavors" and "request_specs" from the "nova_api" database?

Thank you, that's exactly what I wanted to check.

It's not that Horizon didn't pick up the changes, it's that the instances that were launched before those changes were made didn't pick them up, and still run with the old number of vcpus. That is a perfectly normal and expected behavior, as it's impossible to update a running instance. This is also the reason why editing a flavor in Horizon makes it create a new flavor, and mark the old one as deleted, instead of just updating the database.

I'm not sure if there is much we can do in this situation. Of course, it would be much better if Horizon didn't calculate the quotas, but instead ask Nova for them, but that may be impossible to implement in this version, as it would be a new API call.

To sum up, Horizon calculates quotas assuming that all instances were launched with the current set of flavors, while Nova tracks how much resources the actual instances take. The discrepancy is there because some of the flavors were manually changed after launching some instances with them, so they no longer reflect accurately the resources that those instances take up.

The only way to avoid such a situation is to re-launch any instances that use flavors that have been modified, and, preferably, refrain from modifying flavors at all, creating new ones as needed instead.

Hi Radomir,

Thanks a lot for these updates, it's getting clear now. Do you know if in the long term there's a plan to add an API call to Nova to allow Horizon (or other software) to ask about resources consumption?

Cheers,

Seb.

There are plans to rework the quota code on the Horizon side quite extensively ( https://blueprints.launchpad.net/horizon/+spec/make-quotas-great-again ), however I am not entirely sure if this particular problem is going to be addressed by it and how exactly, as I was not involved in the planning. However, those problems have been reported before ( https://bugs.launchpad.net/horizon/+bug/1210533 https://bugs.launchpad.net/horizon/+bug/1277108 ) and I'm sure we will get to them sooner or later.

Hello,

Yesterday we've found the explanation and our customer is now happy with it.
After having done a few more tests, they have closed their support case.

We can close this BZ now.

Cheers,

Seb.

Thank you!