Bug 1154020
Summary: | Unable to access admin project due to 'OverflowError: cannot convert float infinity to integer' error. | ||
---|---|---|---|
Product: | Red Hat OpenStack | Reporter: | Lee Yarwood <lyarwood> |
Component: | python-django-horizon | Assignee: | Julie Pichon <jpichon> |
Status: | CLOSED ERRATA | QA Contact: | Ido Ovadia <iovadia> |
Severity: | high | Docs Contact: | |
Priority: | high | ||
Version: | 5.0 (RHEL 7) | CC: | ajeain, aortega, athomas, dyocum, jpichon, lyarwood, mrunge, pbandark, scohen, slong, yeylon |
Target Milestone: | z3 | Keywords: | Triaged, ZStream |
Target Release: | 5.0 (RHEL 7) | Flags: | ajeain:
needinfo+
|
Hardware: | x86_64 | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | python-django-horizon-2014.1.3-2.el7ost | Doc Type: | Bug Fix |
Doc Text: |
Quota information from other services could previously get out of sync and incorrectly return negative values for "in use" resources. This meant that the dashboard project overview would fail with an error 500. With this update, negative values for "in use" resources are now treated as "zero resources are currently in use", and the dashboard project overview loads correctly for this case.
|
Story Points: | --- |
Clone Of: | Environment: | ||
Last Closed: | 2014-12-02 15:13:49 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Lee Yarwood
2014-10-17 09:31:30 UTC
This looks like https://bugs.launchpad.net/horizon/+bug/1370869 which has a patch (Juno / 6.0), I wonder if it would be worthwhile building a test package that includes it to see if it helps? From the logs it looks like Neutron is also in use, could you have a look at the neutron database too to search for negative quota values? Likewise if Cinder is in use. I'm surprised the quota_usages table doesn't have a field linking it back to a project id, I wonder if the data is actually elsewhere. Does the quota_usages table matches what is shown when running "nova absolute-limits" on the CLI? I'll have a look and investigate which tables actually contain the data Horizon ends up using. (In reply to Julie Pichon from comment #4) > This looks like https://bugs.launchpad.net/horizon/+bug/1370869 which has a > patch (Juno / 6.0), I wonder if it would be worthwhile building a test > package that includes it to see if it helps? Yes please, just to workaround the issue while we look into the root cause. I'll open a separate BZ for this, likely against Nova, once we have more data to go on. > From the logs it looks like Neutron is also in use, could you have a look at > the neutron database too to search for negative quota values? Likewise if > Cinder is in use. I assume that this issue lies on the Nova side when calculating how to populate the quota_usages table. I'm trying to find how this is done now before I escalate to the Nova team. > I'm surprised the quota_usages table doesn't have a field linking it back to > a project id, I wonder if the data is actually elsewhere. Does the > quota_usages table matches what is shown when running "nova absolute-limits" > on the CLI? Actually it does : MariaDB [nova]> describe quota_usages; +---------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +---------------+--------------+------+-----+---------+----------------+ | created_at | datetime | YES | | NULL | | | updated_at | datetime | YES | | NULL | | | deleted_at | datetime | YES | | NULL | | | id | int(11) | NO | PRI | NULL | auto_increment | | project_id | varchar(255) | YES | MUL | NULL | | | resource | varchar(255) | YES | | NULL | | | in_use | int(11) | NO | | NULL | | | reserved | int(11) | NO | | NULL | | | until_refresh | int(11) | YES | | NULL | | | deleted | int(11) | YES | | NULL | | | user_id | varchar(255) | YES | MUL | NULL | | +---------------+--------------+------+-----+---------+----------------+ 11 rows in set (0.00 sec) > I'll have a look and investigate which tables actually contain the data > Horizon ends up using. Thank you for the additional information. I can reproduce the issue locally if I set one of the in_use value to -1 specifically (-2 or -30 is displayed fine on the Overview, but -1 gets converted to "infinity" and that causes the issue). So it seems like there might be another negative quota elsewhere. Could you also provide the output of "cinder --debug absolute-limits" and "neutron quota-show <tenant-id>"? My current suspicion is that the negative value causing the problem may be in the quota_usages table for Cinder. (The upstream fix only corrected for Nova and Neutron doesn't seem to have a usage API.) In case this still doesn't appear to be the issue, could you also provide the /etc/openstack-dashboard/local_settings file for reference? Thank you. # cinder --debug absolute-limits +-------------------------+-------+ | Name | Value | +-------------------------+-------+ | maxTotalSnapshots | 10 | | maxTotalVolumeGigabytes | 1000 | | maxTotalVolumes | 100 | | totalGigabytesUsed | -1 | | totalSnapshotsUsed | 0 | | totalVolumesUsed | -1 | +-------------------------+-------+ After changing those -1 values to 0 the admin project page now loads. MariaDB [cinder]> select * from quota_usages where in_use = '-1'; +---------------------+---------------------+------------+---------+----+----------------------------------+-----------+--------+----------+---------------+ | created_at | updated_at | deleted_at | deleted | id | project_id | resource | in_use | reserved | until_refresh | +---------------------+---------------------+------------+---------+----+----------------------------------+-----------+--------+----------+---------------+ | 2014-09-08 15:47:06 | 2014-10-07 13:36:24 | NULL | 0 | 1 | d1eb069f22fb40fa85578d947ece6ef4 | gigabytes | -1 | 0 | NULL | | 2014-09-08 15:47:06 | 2014-10-07 13:36:24 | NULL | 0 | 2 | d1eb069f22fb40fa85578d947ece6ef4 | volumes | -1 | 0 | NULL | +---------------------+---------------------+------------+---------+----+----------------------------------+-----------+--------+----------+---------------+ 2 rows in set (0.01 sec) MariaDB [cinder]> update quota_usages set in_use = '0' where quota_usages.in_use = '-1' -> ; Query OK, 2 rows affected (0.01 sec) Rows matched: 2 Changed: 2 Warnings: 0 ================== so are we going to fix this issue in new patch / final hotfix pacakages ? Thank you for the update, I will prepare a new patch. How to test: ------------ 1. Consider launching at least 1 instance so that it's easier to edit the database directly 2. Open the nova database and set the in_use value for the resource "instances" to -1 or any negative value, in the quota_usages table 3. Navigate to the overview page for the same tenant you edited: no error 500, it loads fine 4. Consider creating a volume first to make it easier to edit the cinder database directly 5. Open the cinder database and set the in_use value for the resource "volumes" to -1 or any negative value, in the quota_usages table 6. Navigate to the overview page for the same tenant you edited: no error 500, it loads fine *** Bug 1164341 has been marked as a duplicate of this bug. *** Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2014-1929.html |