Bug 1069378
| Summary: | can't launch VM using nova CLI: u'message': u'ImageNotAuthorized'... | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Dan Yocum <dyocum> | ||||||
| Component: | openstack-nova | Assignee: | Dan Smith <dasmith> | ||||||
| Status: | CLOSED NOTABUG | QA Contact: | Ami Jeain <ajeain> | ||||||
| Severity: | high | Docs Contact: | |||||||
| Priority: | high | ||||||||
| Version: | 3.0 | CC: | dasmith, dmaley, dyocum, fpercoco, jdexter, ndipanov, sclewis, sputhenp, yeylon | ||||||
| Target Milestone: | --- | ||||||||
| Target Release: | 5.0 (RHEL 7) | ||||||||
| Hardware: | x86_64 | ||||||||
| OS: | Linux | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2014-03-25 17:34:42 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
Can you add the output of "glance show 8a392883-8798-4385-a71d-0ff9f24a64e5"? there might be some ways to debug it further. 1.Get the glance log file at the time of booting the instance 2. try to run: curl http://hostname:9292/v1/images/<image-id> (HTTP 401) or: curl -H "X-Auth-Token:<token-id>" http://hostname:9292/v1/images/<image-id> In addition, the customer might have fallen into: https://answers.launchpad.net/nova/+question/193324 Dan I'm waiting on CLI access for os1, but from the dashboard and Dan's earlier post I can see from the gss tenant From UI rhel-guest-image-6-6.5-20131115.0-1.qcow2 (fd8c00c8-6c79-4317-8a38-8735e8484064) from CLI (dyocam) rhel-guest-image-6-6.5-20131115.0-1.qcow2 (8a392883-8798-4385-a71d-0ff9f24a64e5) I will confirm once Dan has provided CLI access, as well as logging (In reply to Dan Smith from comment #2) > Can you add the output of "glance show 8a392883-8798-4385-a71d-0ff9f24a64e5"? [root ~]# glance show 8a392883-8798-4385-a71d-0ff9f24a64e5 URI: http://os1-public.osop.rhcloud.com:9292/v1/images/8a392883-8798-4385-a71d-0ff9f24a64e5 Id: 8a392883-8798-4385-a71d-0ff9f24a64e5 Public: Yes Protected: No Name: rhel-guest-image-6-6.5-20131115.0-1.qcow2 Status: active Size: 307962880 Disk format: qcow2 Container format: bare Minimum Ram Required (MB): 0 Minimum Disk Required (GB): 0 Owner: 05c37f247b0c4f6f9682559e17e747ad Created at: 2013-11-22T20:51:43 Updated at: 2013-11-22T20:52:27 Okay, then glance logging is next, I think. The error is actually coming from glance when nova hits it. Also, can you confirm that when you start a guest with Horizon, the image uuid it is started with is the same as the one you're getting the complaint from the CLI about? (In reply to Dan Smith from comment #9) > Okay, then glance logging is next, I think. The error is actually coming > from glance when nova hits it. Also, can you confirm that when you start a > guest with Horizon, the image uuid it is started with is the same as the one > you're getting the complaint from the CLI about? Yeah, using the dashboard absolutely works for these images - and it's not just *this single* image, it's all of them. Can you also post "glance show fd8c00c8-6c79-4317-8a38-8735e8484064"? and just for grins, can you try your nova boot with the uuid of an image instead of a name? For example: nova boot --image 8a392883-8798-4385-a71d-0ff9f24a64e5 --flavor ... You've got at least two images with the same name, so using the image name to boot from the command-line is going to be problematic at best. I expect Horizon looks up the images that you have access to and refers to them by UUID in the boot request. Nova CLI does not (AFAIK). Dan I am attaching 2 files, one is a boot attempt from ui, the other is from the command line. in response to Ami, 2. try to run: curl http://hostname:9292/v1/images/<image-id> (HTTP 401) or: curl -H "X-Auth-Token:<token-id>" http://hostname:9292/v1/images/<image-id> curl -i http://hostname:9292/v1/images/<image-id> results in a 401 error curl -H "X-Auth-Token:<token-id>" http://hostname:9292/v1/images/<image-id> results in a binary file being retrieved I am assuming it is the correct image. Dan I was wrong is on the internal cloud and not on the external fd8c00c8-6c79-4317-8a38-8735e8484064
We get the same error from cli using the image-uuid
[root ~]# nova boot --flavor m1.small --image 8a392883-8798-4385-a71d-0ff9f24a64e5 jdexter-cli-test
+-----------------------------+-------------------------------------------+
| Property | Value |
+-----------------------------+-------------------------------------------+
| status | BUILD |
| updated | 2014-02-25T15:33:36Z |
| OS-EXT-STS:task_state | scheduling |
| key_name | None |
| image | rhel-guest-image-6-6.5-20131115.0-1.qcow2 |
| hostId | |
| OS-EXT-STS:vm_state | building |
| flavor | m1.small |
| id | 69d714f1-4f2e-4d14-869e-bb392d50a2a1 |
| security_groups | [{u'name': u'default'}] |
| user_id | 69416467b9ca4c62bbd249afddfcb5b0 |
| name | jdexter-cli-test |
| adminPass | HftS7szCGr5Y |
| tenant_id | ac0c0175c59942ab8b77c892323f2330 |
| created | 2014-02-25T15:33:36Z |
| OS-DCF:diskConfig | MANUAL |
| metadata | {} |
| accessIPv4 | |
| accessIPv6 | |
| progress | 0 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-AZ:availability_zone | nova |
| config_drive | |
+-----------------------------+-------------------------------------------+
[root ~]# nova show jdexter-cli-test
+-----------------------------+----------------------------------------------------------------------------------------+
| Property | Value |
+-----------------------------+----------------------------------------------------------------------------------------+
| status | ERROR |
| updated | 2014-02-25T15:33:37Z |
| OS-EXT-STS:task_state | None |
| key_name | None |
| image | rhel-guest-image-6-6.5-20131115.0-1.qcow2 (8a392883-8798-4385-a71d-0ff9f24a64e5) |
| hostId | |
| OS-EXT-STS:vm_state | error |
| flavor | m1.small (2) |
| id | 69d714f1-4f2e-4d14-869e-bb392d50a2a1 |
| security_groups | [{u'name': u'default'}] |
| user_id | 69416467b9ca4c62bbd249afddfcb5b0 |
| name | jdexter-cli-test |
| created | 2014-02-25T15:33:36Z |
| fault | {u'message': u'ImageNotAuthorized', u'code': 500, u'created': u'2014-02-25T15:33:37Z'} |
| OS-DCF:diskConfig | MANUAL |
| metadata | {} |
| accessIPv4 | |
| accessIPv6 | |
| tenant_id | ac0c0175c59942ab8b77c892323f2330 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-AZ:availability_zone | nova |
| config_drive |
Created attachment 867518 [details]
glance logs from UI boot
Created attachment 867519 [details]
glance logs from cli boot
>>Are you using the same credentials for the dashboard and the CLI ? -The same credentials are used for dashboard and CLI, affects members as well as the admin user. >> Do you see the same images listed on the dashboard and the CLI? - the same images was used with both dashboard and CLI, when using CLI the UUID was used to boot the instance >>Could you check on the dashboard if the image is flagged as public? This affects booting from any image, but yes the one in the test is public. From the rhos-prio list thread. In addition... 'glance image-list' is failing from the cli, too. 401 - not authorized. Nothing wrong in the keystone log, either. :-( So, now that we've unwitting changed the token_format from UUID to PKI, sync'd /etc/keystone/ssl/* between the control nodes, cleaned up the /var/lib/{nova,glance,cinder}/keystone-signing dirs... things still were NOT working - but, the error was different:
[yocum@gee01 Downloads]$ nova boot --flavor m1.small --image 'rhel-guest-image-6-6.5-20131115.0-1.qcow2' yocum-cli-test-0951
ERROR: [Errno 113] No route to host
wait. what?
[yocum@gee01 Downloads]$ telnet 209.132.178.1 5000
Trying 209.132.178.1...
telnet: connect to address 209.132.178.1: No route to host
seriously?
I looked at iptables - there are nova-api-N chains, now? What's that about, and why is there a multi rule for ports 5000, 35357?
I restarted iptables which cleaned out the rogue multi rule and lo-and-behold - I can telnet to port 5000. I can launch VMs using the CLI from both os1-public.osop.rhcloud.com (external facing controller) and public.os1.phx2.redhat.com (internal facing controller).
It appears to be that toke_format=UUID is really and for true broken in Grizzly wrt glance. Piled on to that, nova-api is mucking about in iptables, which I'm not certain I appreciate.
Let's leave this ticket open for a few days - just to be on the safe side.
Support case was closed at request of customer on 3/14, closing NOTABUG |
Description of problem: Users can't launch VMs using the nova cli. Version-Release number of selected component (if applicable): 3.0 Grizzly How reproducible: Every. Steps to Reproduce: 1. nova boot --flavor m1.small --image 'rhel-guest-image-6-6.5-20131115.0-1.qcow2' yocum-cli-test-1 2. $ nova show yocum-cli-test-1 +-------------------------------------+--------------------------------------------------------------------------------------------------------------------------------+ | Property | Value | +-------------------------------------+--------------------------------------------------------------------------------------------------------------------------------+ | status | ERROR | | updated | 2014-02-24T21:06:14Z | | OS-EXT-STS:task_state | None | | OS-EXT-SRV-ATTR:host | None | | key_name | None | | image | rhel-guest-image-6-6.5-20131115.0-1.qcow2 (8a392883-8798-4385-a71d-0ff9f24a64e5) | | hostId | | | OS-EXT-STS:vm_state | error | | OS-EXT-SRV-ATTR:instance_name | instance-000031b1 | | OS-EXT-SRV-ATTR:hypervisor_hostname | None | | flavor | m1.small (2) | | id | d646de23-b96c-43b4-9f20-cdd1e00daf4d | | security_groups | [{u'name': u'default'}] | | user_id | 2ba0e555cd2e45e0a841ce58e24ef4b4 | | name | yocum-cli-test-1 | | created | 2014-02-24T21:06:13Z | | tenant_id | 05c37f247b0c4f6f9682559e17e747ad | | OS-DCF:diskConfig | MANUAL | | metadata | {} | | accessIPv4 | | | accessIPv6 | | | fault | {u'message': u'ImageNotAuthorized', u'code': 500, u'details': u'Not authorized for image 8a392883-8798-4385-a71d-0ff9f24a64e5. | | | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 224, in decorated_function | | | return function(self, context, *args, **kwargs) | | | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1240, in run_instance | | | do_run_instance() | | | File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 242, in inner | | | retval = f(*args, **kwargs) | | | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1239, in do_run_instance | | | admin_password, is_first_time, node, instance) | | | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 895, in _run_instance | | | self._set_instance_error_state(context, instance[\'uuid\']) | | | File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__ | | | self.gen.next() | | | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 816, in _run_instance | | | image_meta = self._check_image_size(context, instance) | | | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1022, in _check_image_size | | | image_meta = _get_image_meta(context, instance[\'image_ref\']) | | | File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 268, in _get_image_meta | | | return image_service.show(context, image_id) | | | File "/usr/lib/python2.6/site-packages/nova/image/glance.py", line 239, in show | | | _reraise_translated_image_exception(image_id) | | | File "/usr/lib/python2.6/site-packages/nova/image/glance.py", line 237, in show | | | image = self._client.call(context, 1, \'get\', image_id) | | | File "/usr/lib/python2.6/site-packages/nova/image/glance.py", line 182, in call | | | return getattr(client.images, method)(*args, **kwargs) | | | File "/usr/lib/python2.6/site-packages/glanceclient/v1/images.py", line 104, in get | | | % urllib.quote(image_id)) | | | File "/usr/lib/python2.6/site-packages/glanceclient/common/http.py", line 245, in raw_request | | | return self._http_request(url, method, **kwargs) | | | File "/usr/lib/python2.6/site-packages/glanceclient/common/http.py", line 206, in _http_request | | | raise exc.from_response(resp, body_str) | | | ', u'created': u'2014-02-24T21:06:14Z'} | | OS-EXT-STS:power_state | 0 | | OS-EXT-AZ:availability_zone | nova | | config_drive | | +-------------------------------------+--------------------------------------------------------------------------------------------------------------------------------+