Description of problem: I deploy one node openstack(two nic, neutron gre) with RDO, and the answer-file is :http://paste.openstack.org/show/58484/ then I do a simple autoscaling with neutron lb,the template is : http://paste.openstack.org/show/58486/ when I stress the vm , it got notifer in ceilometer-alarm but in heat-engine.log, got authentiaion errors: http://paste.openstack.org/show/58506/ Version-Release number of selected component (if applicable): [root@gre1 heat(keystone_admin)]# rpm -qa | grep openstack openstack-heat-engine-2013.2.1-1.0.el6.noarch openstack-ceilometer-central-2013.2.1-1.el6.noarch openstack-ceilometer-collector-2013.2.1-1.el6.noarch openstack-nova-conductor-2013.2.1-1.el6.noarch openstack-nova-cert-2013.2.1-1.el6.noarch openstack-neutron-openvswitch-2013.2.1-1.el6.noarch openstack-keystone-2013.2.1-1.el6.noarch openstack-utils-2013.2-2.el6.noarch openstack-nova-common-2013.2.1-1.el6.noarch openstack-ceilometer-compute-2013.2.1-1.el6.noarch openstack-heat-common-2013.2.1-1.0.el6.noarch openstack-heat-api-2013.2.1-1.0.el6.noarch openstack-heat-api-cfn-2013.2.1-1.0.el6.noarch openstack-ceilometer-alarm-2013.2.1-1.el6.noarch openstack-ceilometer-api-2013.2.1-1.el6.noarch openstack-nova-console-2013.2.1-1.el6.noarch openstack-nova-compute-2013.2.1-1.el6.noarch openstack-nova-scheduler-2013.2.1-1.el6.noarch openstack-dashboard-2013.2.1-1.el6.noarch openstack-selinux-0.1.3-2.el6ost.noarch openstack-heat-api-cloudwatch-2013.2.1-1.0.el6.noarch openstack-glance-2013.2-1.el6.noarch openstack-nova-api-2013.2.1-1.el6.noarch openstack-ceilometer-common-2013.2.1-1.el6.noarch openstack-packstack-2013.2.1-0.25.dev936.el6.noarch openstack-nova-novncproxy-2013.2.1-1.el6.noarch openstack-neutron-2013.2.1-1.el6.noarch python-django-openstack-auth-1.1.2-1.el6.noarch How reproducible: Steps to Reproduce: 1.deploy one node openstack(neutron, gre) 2.create autoscaling stack Actual results: the scalup policy is trigger but no more instance is launched Expected results: Additional info:
HI , after I reinstall my openstack with allinone, answer-file: http://paste.openstack.org/show/58919/ got the same promble.
Angus, Any ideas on the authentication problems?
Note: the signed url has been validated, the problem is in getting the availability zone in the template (this is on initial loading of the stack). I haven't seen this before. It might be worth asking shardy about the v2 auth. 2013-12-30 19:03:19.062 2608 TRACE heat.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/heat/engine/template.py", line 119, in handle_get_az 2013-12-30 19:03:19.062 2608 TRACE heat.openstack.common.rpc.amqp return stack.get_availability_zones() 2013-12-30 19:03:19.062 2608 TRACE heat.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/heat/engine/parser.py", line 641, in get_availability_zones 2013-12-30 19:03:19.062 2608 TRACE heat.openstack.common.rpc.amqp self.clients.nova().availability_zones.list(detailed=False)] 2013-12-30 19:03:19.062 2608 TRACE heat.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/heat/engine/clients.py", line 93, in nova 2013-12-30 19:03:19.062 2608 TRACE heat.openstack.common.rpc.amqp if self.auth_token is None: 2013-12-30 19:03:19.062 2608 TRACE heat.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/heat/engine/clients.py", line 76, in auth_token 2013-12-30 19:03:19.062 2608 TRACE heat.openstack.common.rpc.amqp return self.context.auth_token or self.keystone().auth_token 2013-12-30 19:03:19.062 2608 TRACE heat.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/heat/common/heat_keystoneclient.py", line 313, in auth_token 2013-12-30 19:03:19.062 2608 TRACE heat.openstack.common.rpc.amqp return self.client_v2.auth_token 2013-12-30 19:03:19.062 2608 TRACE heat.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/heat/common/heat_keystoneclient.py", line 73, in client_v2 2013-12-30 19:03:19.062 2608 TRACE heat.openstack.common.rpc.amqp self._client_v2 = self._v2_client_init() 2013-12-30 19:03:19.062 2608 TRACE heat.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/heat/common/heat_keystoneclient.py", line 103, in _v2_client_init 2013-12-30 19:03:19.062 2608 TRACE heat.openstack.common.rpc.amqp client_v2 = kc.Client(**kwargs) 2013-12-30 19:03:19.062 2608 TRACE heat.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 139, in __init__ 2013-12-30 19:03:19.062 2608 TRACE heat.openstack.common.rpc.amqp self.authenticate() 2013-12-30 19:03:19.062 2608 TRACE heat.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/keystoneclient/httpclient.py", line 468, in authenticate 2013-12-30 19:03:19.062 2608 TRACE heat.openstack.common.rpc.amqp resp, body = self.get_raw_token_from_identity_service(**kwargs) 2013-12-30 19:03:19.062 2608 TRACE heat.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 162, in get_raw_token_from_identity_service 2013-12-30 19:03:19.062 2608 TRACE heat.openstack.common.rpc.amqp token=token) 2013-12-30 19:03:19.062 2608 TRACE heat.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 197, in _base_authN 2013-12-30 19:03:19.062 2608 TRACE heat.openstack.common.rpc.amqp resp, body = self.request(url, 'POST', body=params, headers=headers) 2013-12-30 19:03:19.062 2608 TRACE heat.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/keystoneclient/httpclient.py", line 610, in request 2013-12-30 19:03:19.062 2608 TRACE heat.openstack.common.rpc.amqp **request_kwargs) 2013-12-30 19:03:19.062 2608 TRACE heat.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/keystoneclient/httpclient.py", line 124, in request 2013-12-30 19:03:19.062 2608 TRACE heat.openstack.common.rpc.amqp raise exceptions.from_response(resp, method, url) 2013-12-30 19:03:19.062 2608 TRACE heat.openstack.common.rpc.amqp Unauthorized: The request you have made requires authentication. (HTTP 401) 2013-12-30 19:03:19.062 2608 TRACE heat.openstack.common.rpc.amqp
Created attachment 844125 [details] yaml template that results in authentication error
Created attachment 844126 [details] Answer file from packstack
Steve, Any thoughts on the auth going wrong here?
(In reply to Steven Dake from comment #6) > Steve, > > Any thoughts on the auth going wrong here? I've not managed to reproduce yet - Did you reproduce the issue with the template attached to comment #4? The issue is we're failing to get a token from keystone with the stored credentials, prior to requesting the list of availability zones from nova, I just need to figure out what configuration causes the scenario reported.
shardy, I didn't actually try since I was unplugged from work for the shutdown. The reporter indicated it seems to happen under heavy load. It wasn't clear if heavy load was required to reproduce the issue. Alienyyg, Would you mind clarifying if the problem was intermittent and only occurred under heavy load, or always happened in your setup?
(In reply to Steven Dake from comment #8) > shardy, > > I didn't actually try since I was unplugged from work for the shutdown. The > reporter indicated it seems to happen under heavy load. It wasn't clear if > heavy load was required to reproduce the issue. > > Alienyyg, > > Would you mind clarifying if the problem was intermittent and only occurred > under heavy load, or always happened in your setup? steven: sorry fot this, but I have delete this openstack because of limited resource:( Regards
Alienyyg, There is no problem that the OpenStack instance has been deleted. I am curious however if the problem was consistent and happened every time, or only under heavy load. If you don't recall, that is fine, just let us know since more information would help us reproduce this problem. Regards -steve
(In reply to Steven Dake from comment #10) > Alienyyg, > > There is no problem that the OpenStack instance has been deleted. I am > curious however if the problem was consistent and happened every time, or > only under heavy load. If you don't recall, that is fine, just let us know > since more information would help us reproduce this problem. > > Regards > -steve steve: after the deploy finished, and I can launch a isntance without any error, I change the interval in pipline.yaml ,then restart all the ceilometer service,then I launch that stack, after the stack finshed ,then install stress to the webserver instance,then start to stress the instance. I have a question: this is a autoscaling stack ,so if I do not stress the instance to heavey load, the ceilometer notifer won't be triggered so does the error in heat-engine,right? if so, this can only happen in heavey load. on the other hand,the log information include the create of the stack, so it the authentication always appears, maybe heat can not even create the webserver instance,right? forgive me but I can only remember this, hope it will help. regards alienyyg
Hi : I also met this problem when I use nova network and aws compatible resource for autoscaling, and the keystone.log are: http://paste.openstack.org/show/59842/ my keystone.config is: http://paste.openstack.org/show/59843/ regards alienyyg
So, after some investigation, I cannot reproduce any problems when the stack in up and in CREATE_COMPLETE state, however I did notice that this error can happen when you delete the stack, if either some watch data, a signal or a periodic task tries to create a heat_keystoneclient.KeystoneClient() object We can probably do better in terms of suppressing this error when the delete is in progress, I'll further investigate the situations where this can happen and raise an upstream bug. I've so far been unable to reproduce the reporters issue, on scale up and scale down all works OK for me, but I've only tested with the CW api not ceilometer. I'm setting up a second test platform with a fresh Havana install with neutron ceilometer so I can attempt a reproduce with an environment closer to the reporter.
hi all: I have finished the autoscaling with aws compatible resources,the authentication error is cause by the password of admin,I launch a stack via dashboard, but my broswer remember a legacy password, and I forgot this :( sorry for the needless work for this "bug" Best Regards alienyyg