Description of problem: Unable to access admin project due to 'OverflowError: cannot convert float infinity to integer' error. Access to all other projects works at the present time. We see the following trace in the logs : 2014-10-07 15:06:03,565 6144 DEBUG horizon.base Panel with slug "loadbalancers" is not registered with Dashboard "project". 2014-10-07 15:06:03,566 6144 DEBUG horizon.base Panel with slug "firewalls" is not registered with Dashboard "project". 2014-10-07 15:06:03,566 6144 DEBUG horizon.base Panel with slug "vpn" is not registered with Dashboard "project". 2014-10-07 15:06:03,566 6144 DEBUG horizon.base Panel with slug "loadbalancers" is not registered with Dashboard "project". 2014-10-07 15:06:03,566 6144 DEBUG horizon.base Panel with slug "firewalls" is not registered with Dashboard "project". 2014-10-07 15:06:03,566 6144 DEBUG horizon.base Panel with slug "vpn" is not registered with Dashboard "project". 2014-10-07 15:06:03,600 6144 ERROR django.request Internal Server Error: /dashboard/project/ Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 137, in get_response response = response.render() File "/usr/lib/python2.7/site-packages/django/template/response.py", line 105, in render self.content = self.rendered_content File "/usr/lib/python2.7/site-packages/django/template/response.py", line 82, in rendered_content content = template.render(context) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 140, in render return self._render(context) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 134, in _render return self.nodelist.render(context) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 840, in render bit = self.render_node(node, context) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 854, in render_node return node.render(context) File "/usr/lib/python2.7/site-packages/django/template/loader_tags.py", line 123, in render return compiled_parent._render(context) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 134, in _render return self.nodelist.render(context) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 840, in render bit = self.render_node(node, context) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 854, in render_node return node.render(context) File "/usr/lib/python2.7/site-packages/django/template/loader_tags.py", line 62, in render result = block.nodelist.render(context) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 840, in render bit = self.render_node(node, context) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 854, in render_node return node.render(context) File "/usr/lib/python2.7/site-packages/django/template/loader_tags.py", line 62, in render result = block.nodelist.render(context) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 840, in render bit = self.render_node(node, context) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 854, in render_node return node.render(context) File "/usr/lib/python2.7/site-packages/django/template/loader_tags.py", line 155, in render return self.render_template(self.template, context) File "/usr/lib/python2.7/site-packages/django/template/loader_tags.py", line 137, in render_template output = template.render(context) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 140, in render return self._render(context) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 134, in _render return self.nodelist.render(context) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 840, in render bit = self.render_node(node, context) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 854, in render_node return node.render(context) File "/usr/lib/python2.7/site-packages/django/template/defaulttags.py", line 305, in render return nodelist.render(context) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 840, in render bit = self.render_node(node, context) File "/usr/lib/python2.7/site-packages/django/template/base.py", line 854, in render_node return node.render(context) File "/usr/lib/python2.7/site-packages/django/template/defaulttags.py", line 488, in render return str(int(round(ratio))) OverflowError: cannot convert float infinity to integer Version-Release number of selected component (if applicable): python-django-horizon-2014.1.2-2.el7ost.noarch How reproducible: Always. Steps to Reproduce: 1. Attempt to access the Admin project via Horizon. Actual results: Unable to access the admin project due to a 500 internal server error. Expected results: Able to access the admin project. Additional info: This appeared to be a hit of the U/S bug #1268573 [1] however the issue remains after resetting the negative floating_ips quota value in the DB. [1] https://bugs.launchpad.net/horizon/+bug/1268573
This looks like https://bugs.launchpad.net/horizon/+bug/1370869 which has a patch (Juno / 6.0), I wonder if it would be worthwhile building a test package that includes it to see if it helps? From the logs it looks like Neutron is also in use, could you have a look at the neutron database too to search for negative quota values? Likewise if Cinder is in use. I'm surprised the quota_usages table doesn't have a field linking it back to a project id, I wonder if the data is actually elsewhere. Does the quota_usages table matches what is shown when running "nova absolute-limits" on the CLI? I'll have a look and investigate which tables actually contain the data Horizon ends up using.
(In reply to Julie Pichon from comment #4) > This looks like https://bugs.launchpad.net/horizon/+bug/1370869 which has a > patch (Juno / 6.0), I wonder if it would be worthwhile building a test > package that includes it to see if it helps? Yes please, just to workaround the issue while we look into the root cause. I'll open a separate BZ for this, likely against Nova, once we have more data to go on. > From the logs it looks like Neutron is also in use, could you have a look at > the neutron database too to search for negative quota values? Likewise if > Cinder is in use. I assume that this issue lies on the Nova side when calculating how to populate the quota_usages table. I'm trying to find how this is done now before I escalate to the Nova team. > I'm surprised the quota_usages table doesn't have a field linking it back to > a project id, I wonder if the data is actually elsewhere. Does the > quota_usages table matches what is shown when running "nova absolute-limits" > on the CLI? Actually it does : MariaDB [nova]> describe quota_usages; +---------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +---------------+--------------+------+-----+---------+----------------+ | created_at | datetime | YES | | NULL | | | updated_at | datetime | YES | | NULL | | | deleted_at | datetime | YES | | NULL | | | id | int(11) | NO | PRI | NULL | auto_increment | | project_id | varchar(255) | YES | MUL | NULL | | | resource | varchar(255) | YES | | NULL | | | in_use | int(11) | NO | | NULL | | | reserved | int(11) | NO | | NULL | | | until_refresh | int(11) | YES | | NULL | | | deleted | int(11) | YES | | NULL | | | user_id | varchar(255) | YES | MUL | NULL | | +---------------+--------------+------+-----+---------+----------------+ 11 rows in set (0.00 sec) > I'll have a look and investigate which tables actually contain the data > Horizon ends up using.
Thank you for the additional information. I can reproduce the issue locally if I set one of the in_use value to -1 specifically (-2 or -30 is displayed fine on the Overview, but -1 gets converted to "infinity" and that causes the issue). So it seems like there might be another negative quota elsewhere. Could you also provide the output of "cinder --debug absolute-limits" and "neutron quota-show <tenant-id>"? My current suspicion is that the negative value causing the problem may be in the quota_usages table for Cinder. (The upstream fix only corrected for Nova and Neutron doesn't seem to have a usage API.) In case this still doesn't appear to be the issue, could you also provide the /etc/openstack-dashboard/local_settings file for reference? Thank you.
# cinder --debug absolute-limits +-------------------------+-------+ | Name | Value | +-------------------------+-------+ | maxTotalSnapshots | 10 | | maxTotalVolumeGigabytes | 1000 | | maxTotalVolumes | 100 | | totalGigabytesUsed | -1 | | totalSnapshotsUsed | 0 | | totalVolumesUsed | -1 | +-------------------------+-------+ After changing those -1 values to 0 the admin project page now loads. MariaDB [cinder]> select * from quota_usages where in_use = '-1'; +---------------------+---------------------+------------+---------+----+----------------------------------+-----------+--------+----------+---------------+ | created_at | updated_at | deleted_at | deleted | id | project_id | resource | in_use | reserved | until_refresh | +---------------------+---------------------+------------+---------+----+----------------------------------+-----------+--------+----------+---------------+ | 2014-09-08 15:47:06 | 2014-10-07 13:36:24 | NULL | 0 | 1 | d1eb069f22fb40fa85578d947ece6ef4 | gigabytes | -1 | 0 | NULL | | 2014-09-08 15:47:06 | 2014-10-07 13:36:24 | NULL | 0 | 2 | d1eb069f22fb40fa85578d947ece6ef4 | volumes | -1 | 0 | NULL | +---------------------+---------------------+------------+---------+----+----------------------------------+-----------+--------+----------+---------------+ 2 rows in set (0.01 sec) MariaDB [cinder]> update quota_usages set in_use = '0' where quota_usages.in_use = '-1' -> ; Query OK, 2 rows affected (0.01 sec) Rows matched: 2 Changed: 2 Warnings: 0 ================== so are we going to fix this issue in new patch / final hotfix pacakages ?
Thank you for the update, I will prepare a new patch.
How to test: ------------ 1. Consider launching at least 1 instance so that it's easier to edit the database directly 2. Open the nova database and set the in_use value for the resource "instances" to -1 or any negative value, in the quota_usages table 3. Navigate to the overview page for the same tenant you edited: no error 500, it loads fine 4. Consider creating a volume first to make it easier to edit the cinder database directly 5. Open the cinder database and set the in_use value for the resource "volumes" to -1 or any negative value, in the quota_usages table 6. Navigate to the overview page for the same tenant you edited: no error 500, it loads fine
*** Bug 1164341 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2014-1929.html