Bug 1294469

Summary: nova.api.openstack.extensions Unexpected exception in API method The resource could not be found
Product: Red Hat OpenStack Reporter: Ronnie Rasouli <rrasouli>
Component: openstack-novaAssignee: Sven Anderson <svanders>
Status: CLOSED NOTABUG QA Contact: nlevinki <nlevinki>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 8.0 (Liberty)CC: berrange, dasmith, eglynn, jdonohue, kchamart, mbooth, robert.h.armstrong, robert.w.love, rrasouli, sasha, sbauza, sferdjao, sgordon, shivapriya.o.hiremath, srevivo, svanders, vromanso
Target Milestone: ---Keywords: BuildBlocker, Reopened, ZStream
Target Release: 8.0 (Liberty)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-01-08 14:11:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
nova api-log from the undercloud none

Description Ronnie Rasouli 2015-12-28 11:27:12 UTC 
Created attachment 1109998 [details]
nova api-log from the undercloud

Description of problem:

Deployed RHOS8 with ospd 

openstack overcloud deploy --templates ~/templates/my-overcloud   --control-scale 3 --compute-scale 1 --ntp-server clock.redhat.com   --libvirt-type qemu -e ~/templates/my-overcloud/environments/network-isolation.yaml  -e ~/templates/network-environment.yaml  -e ~/templates/firstboot-environment.yaml

nova list
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                    | Status | Task State | Power State | Networks            |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| e89dffe3-f499-4589-8da6-e7069d34f1fc | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.0.2.10 |
| b68c708c-2105-4dec-8fbb-a1419f44a071 | overcloud-controller-1  | ACTIVE | -          | Running     | ctlplane=192.0.2.7  |
| a4065dcc-6b2b-4210-b005-7f2d8c78fc7a | overcloud-controller-2  | ACTIVE | -          | Running     | ctlplane=192.0.2.8  |
| df8d71d1-37d3-4123-b1f0-36730badf897 | overcloud-novacompute-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.9  |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+


Deployment failed in post deployment 

[root@instack ~]# heat resource-list overcloud | grep -iv complete
+-------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+---------------------+
| resource_name                             | physical_resource_id                          | resource_type                                     | resource_status | updated_time        |
+-------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+---------------------+
| ComputeNodesPostDeployment                | 19e6d2d9-3516-4b9b-8b00-4d467aa0a483          | OS::TripleO::ComputePostDeployment                | CREATE_FAILED   | 2015-12-28T08:37:10 |
| ControllerNodesPostDeployment             | cb388eb8-93d5-49c8-85cc-d097154ae4a0          | OS::TripleO::ControllerPostDeployment             | CREATE_FAILED   | 2015-12-28T08:37:10 |
+-------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+---------------------+

| resource_status        | CREATE_FAILED                                                                                                                                                                                                       |
| resource_status_reason | resources.ControllerOvercloudServicesDeployment_Step4: resources.ControllerNodesPostDeployment.Error: resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6 |
| resource_type          | OS::TripleO::ControllerPostDeployment                                                                                                                          
in nova log in undercloud

'user_name': u'ceilometer'} enforce /usr/lib/python2.7/site-packages/nova/policy.py:104
2015-12-27 10:35:44.284 29993 DEBUG neutronclient.v2_0.client [req-970e33c5-3ae6-4e3d-85f9-2e0a19fb7c13 dab83ae3d9da4af496473e9a5a7c5e4b 26c7d2545a1a416d847e4c9b1b49f8b9 - - -] Error message: 404 Not Found
2015-12-27 10:35:44.284 29993 ERROR nova.api.openstack.extensions [req-970e33c5-3ae6-4e3d-85f9-2e0a19fb7c13 dab83ae3d9da4af496473e9a5a7c5e4b 26c7d2545a1a416d847e4c9b1b49f8b9 - - -] Unexpected exception in API method
2015-12-27 10:35:44.284 29993 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2015-12-27 10:35:44.284 29993 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2015-12-27 10:35:44.284 29993 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2015-12-27 10:35:44.284 29993 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/floating_ips.py", line 117, in index
2015-12-27 10:35:44.284 29993 ERROR nova.api.openstack.extensions     floating_ips = self.network_api.get_floating_ips_by_project(context)
2015-12-27 10:35:44.284 29993 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1402, in get_floating_ips_by_project
2015-12-27 10:35:44.284 29993 ERROR nova.api.openstack.extensions     fips = client.list_floatingips(tenant_id=project_id)['floatingips']
2015-12-27 10:35:44.284 29993 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 102, in with_params
2015-12-27 10:35:44.284 29993 ERROR nova.api.openstack.extensions     ret = self.function(instance, *args, **kwargs)
2015-12-27 10:35:44.284 29993 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 731, in list_floatingips

in nova --debug list the resource could not be found

DEBUG (v2:62) Making authentication request to http://192.0.2.1:5000/v2.0/tokens
DEBUG (connectionpool:383) "POST /v2.0/tokens HTTP/1.1" 200 3493
DEBUG (session:195) REQ: curl -g -i -X GET http://192.0.2.1:8774/v2/26c7d2545a1a416d847e4c9b1b49f8b9 -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}3af214b1c154d65381c533e50a5b1b30a2a64b32"
INFO (connectionpool:203) Starting new HTTP connection (1): 192.0.2.1
DEBUG (connectionpool:383) "GET /v2/26c7d2545a1a416d847e4c9b1b49f8b9 HTTP/1.1" 404 52
DEBUG (session:224) RESP: [404] date: Mon, 28 Dec 2015 11:15:16 GMT connection: keep-alive content-type: text/plain; charset=UTF-8 content-length: 52 x-compute-request-id: req-bcbd6d95-e653-4416-8f53-5f0c1948fc8e 
RESP BODY: 404 Not Found

The resource could not be found.


Version-Release number of selected component (if applicable):

python-nova-12.0.0-4.el7ost.noarch
openstack-nova-common-12.0.0-4.el7ost.noarch
python-novaclient-3.1.0-2.el7ost.noarch
openstack-nova-scheduler-12.0.0-4.el7ost.noarch
openstack-nova-cert-12.0.0-4.el7ost.noarch
openstack-nova-conductor-12.0.0-4.el7ost.noarch
openstack-nova-compute-12.0.0-4.el7ost.noarch
openstack-nova-api-12.0.0-4.el7ost.noarch


How reproducible:
100%

Steps to Reproduce:
1. deploy OSPd version 8 with network isolation and a first-boot script(available in http://file.tlv.redhat.com/~rrasouli/scripts/)

2. Deployment fails with non zero (6)

Actual results:

post deployment failed

Expected results:

success in post deployment

Additional info:

in nova-api.log in controllers can see a version mismatch alert
2015-12-28 03:48:58.369 3843 ERROR nova Traceback (most recent call last):
2015-12-28 03:48:58.369 3843 ERROR nova   File "/usr/bin/nova-manage", line 10, in <module>
2015-12-28 03:48:58.369 3843 ERROR nova     sys.exit(main())
2015-12-28 03:48:58.369 3843 ERROR nova   File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 1443, in main
2015-12-28 03:48:58.369 3843 ERROR nova     ret = fn(*fn_args, **fn_kwargs)
2015-12-28 03:48:58.369 3843 ERROR nova   File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 910, in sync
2015-12-28 03:48:58.369 3843 ERROR nova     return migration.db_sync(version)
2015-12-28 03:48:58.369 3843 ERROR nova   File "/usr/lib/python2.7/site-packages/nova/db/migration.py", line 26, in db_sync
2015-12-28 03:48:58.369 3843 ERROR nova     return IMPL.db_sync(version=version, database=database)
2015-12-28 03:48:58.369 3843 ERROR nova   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/migration.py", line 106, in db_sync
2015-12-28 03:48:58.369 3843 ERROR nova     version)
2015-12-28 03:48:58.369 3843 ERROR nova   File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgrade
2015-12-28 03:48:58.369 3843 ERROR nova     return _migrate(url, repository, version, upgrade=True, err=err, **opts)
2015-12-28 03:48:58.369 3843 ERROR nova   File "<string>", line 2, in _migrate
2015-12-28 03:48:58.369 3843 ERROR nova   File "/usr/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 160, in with_engine
2015-12-28 03:48:58.369 3843 ERROR nova     return f(*a, **kw)
2015-12-28 03:48:58.369 3843 ERROR nova   File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 366, in _migrate
2015-12-28 03:48:58.369 3843 ERROR nova     schema.runchange(ver, change, changeset.step)
2015-12-28 03:48:58.369 3843 ERROR nova   File "/usr/lib/python2.7/site-packages/migrate/versioning/schema.py", line 91, in runchange
2015-12-28 03:48:58.369 3843 ERROR nova     (self.version, startver))
2015-12-28 03:48:58.369 3843 ERROR nova InvalidVersionError: 239 is not 232




nova-api log
Comment 2 Alexander Chuzhoy 2015-12-28 20:53:10 UTC 
Probably a duplicate of BZ #1281613
Comment 3 shivapriya.o.hiremath 2016-10-26 23:09:04 UTC 
Is this issue resolved in OSP 8 ? Because I see the same issue in my deployment of OSP 8. Appreciate any inputs on how to get past this.
Comment 4 Christopher Brown 2017-02-15 17:27:01 UTC 
I'm not seeing this any longer in Newton / OSP 10.
Comment 5 Sven Anderson 2017-05-17 13:02:11 UTC 
Can we close this?
Comment 6 Ronnie Rasouli 2017-05-18 07:19:34 UTC 
I guess so as EOL or current release
Comment 7 Matthew Booth 2018-01-08 14:11:57 UTC 
Seems to have been reopened accidentally? If it was deliberate, please leave a comment.