Bug 1427153 - Failed to abort overcloud update
Summary: Failed to abort overcloud update
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-common
Version: 11.0 (Ocata)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z4
: 11.0 (Ocata)
Assignee: Brad P. Crochet
QA Contact: Yurii Prokulevych
URL:
Whiteboard:
Depends On:
Blocks: 1394025
TreeView+ depends on / blocked
 
Reported: 2017-02-27 13:32 UTC by Yurii Prokulevych
Modified: 2023-02-22 23:02 UTC (History)
13 users (show)

Fixed In Version: openstack-tripleo-common-6.1.1-1.el7ost python-tripleoclient-6.2.0-2.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-01-26 15:50:49 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1668269 0 None None None 2017-02-27 13:45:50 UTC
OpenStack gerrit 481673 0 None MERGED Remove update abort 2020-04-14 15:05:14 UTC
OpenStack gerrit 481679 0 None MERGED Remove update abort 2020-04-14 15:05:14 UTC

Description Yurii Prokulevych 2017-02-27 13:32:39 UTC
Description of problem:
-----------------------
Attempt to abort overcloud update hangs:

overcloud update abort overcloud
Started Mistral Workflow tripleo.package_update.v1.cancel_stack_update. Execution ID: d732cefd-9ade-48d2-9c39-0a5c60d092f0
Waiting for messages on queue '66e08597-312c-4574-ac93-3f3bac79c778' with no timeout.

Looks like mistral tasks fail:
------------------------------
mistral task-list d732cefd-9ade-48d2-9c39-0a5c60d092f0
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for 192.168.24.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for 192.168.24.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
+------------------------------+------------------------------+------------------------------+------------------------------+---------+------------------------------+---------------------+---------------------+
| ID                           | Name                         | Workflow name                | Execution ID                 | State   | State info                   | Created at          | Updated at          |
+------------------------------+------------------------------+------------------------------+------------------------------+---------+------------------------------+---------------------+---------------------+
| 416e7e3c-5502-4eb3-bf31-f6d4 | set_cancel_stack_update_fail | tripleo.package_update.v1.ca | d732cefd-9ade-               | SUCCESS | None                         | 2017-02-27 13:04:46 | 2017-02-27 13:04:46 |
| 9b61c298                     | ed                           | ncel_stack_update            | 48d2-9c39-0a5c60d092f0       |         |                              |                     |                     |
| d6c3ea0e-                    | send_message                 | tripleo.package_update.v1.ca | d732cefd-9ade-               | ERROR   | Failed to run action [act... | 2017-02-27 13:04:46 | 2017-02-27 13:04:47 |
| eaf7-4924-8233-18e4030770ab  |                              | ncel_stack_update            | 48d2-9c39-0a5c60d092f0       |         |                              |                     |                     |
| fa41b255-382b-44ee-          | cancel_stack_update          | tripleo.package_update.v1.ca | d732cefd-9ade-               | ERROR   | Failed to run action [act... | 2017-02-27 13:04:46 | 2017-02-27 13:04:46 |
| bde2-31b197fb2789            |                              | ncel_stack_update            | 48d2-9c39-0a5c60d092f0       |         |                              |                     |                     |
+------------------------------+------------------------------+------------------------------+------------------------------+---------+------------------------------+---------------------+---------------------+

mistral task-get 416e7e3c-5502-4eb3-bf31-f6d49b61c298
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for 192.168.24.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for 192.168.24.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
+---------------+-----------------------------------------------+
| Field         | Value                                         |
+---------------+-----------------------------------------------+
| ID            | 416e7e3c-5502-4eb3-bf31-f6d49b61c298          |
| Name          | set_cancel_stack_update_failed                |
| Workflow name | tripleo.package_update.v1.cancel_stack_update |
| Execution ID  | d732cefd-9ade-48d2-9c39-0a5c60d092f0          |
| State         | SUCCESS                                       |
| State info    | None                                          |
| Created at    | 2017-02-27 13:04:46                           |
| Updated at    | 2017-02-27 13:04:46                           |
+---------------+-----------------------------------------------+

mistral task-get d6c3ea0e-eaf7-4924-8233-18e4030770ab
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for 192.168.24.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for 192.168.24.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field         | Value                                                                                                                                                                                           |
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ID            | d6c3ea0e-eaf7-4924-8233-18e4030770ab                                                                                                                                                            |
| Name          | send_message                                                                                                                                                                                    |
| Workflow name | tripleo.package_update.v1.cancel_stack_update                                                                                                                                                   |
| Execution ID  | d732cefd-9ade-48d2-9c39-0a5c60d092f0                                                                                                                                                            |
| State         | ERROR                                                                                                                                                                                           |
| State info    | Failed to run action [action_ex_id=394def26-17d3-44e6-b99f-65751adc9813, action_cls='<class 'mistral.actions.action_factory.ZaqarAction'>', attributes='{u'client_method_name':                 |
|               | u'queue_post'}', params='{u'queue_name': u'<% $.queue_name', u'messages': {u'body': {u'type': u'tripleo.package_update.v1.cancel_stack_update', u'payload': {u'status': u'FAILED', u'message':  |
|               | u"Failed to run action [action_ex_id=09e4ea8b-54b1-4944-8e89-1db3de529512, action_cls='<class 'mistral.actions.action_factory.CancelStackUpdateAction'>', attributes='{}',                      |
|               | params='{u'stack_id': u'5f54018a-938d-4503-ac21-634b9b7ff37a'}']\n __init__() takes exactly 5 arguments (4 given)", u'execution': {u'input': {u'stack_id': u'5f54018a-                          |
|               | 938d-4503-ac21-634b9b7ff37a', u'queue_name': u'66e08597-312c-4574-ac93-3f3bac79c778'}, u'params': {}, u'id': u'd732cefd-9ade-48d2-9c39-0a5c60d092f0', u'name':                                  |
|               | u'tripleo.package_update.v1.cancel_stack_update', u'spec': {u'input': [u'stack_id', {u'queue_name': u'tripleo'}], u'tasks': {u'cancel_stack_update': {u'name': u'cancel_stack_update', u'on-    |
|               | error': u'set_cancel_stack_update_failed', u'on-success': u'send_message', u'version': u'2.0', u'action': u'tripleo.package_update.cancel_stack_update stack_id=<% $.stack_id %>', u'type':     |
|               | u'direct'}, u'send_message': {u'action': u'zaqar.queue_post', u'input': {u'queue_name': u'<% $.queue_name', u'messages': {u'body': {u'type': u'tripleo.package_update.v1.cancel_stack_update',  |
|               | u'payload': {u'status': u"<% $.get('status', 'SUCCESS') %>", u'message': u"<% $.get('message', '') %>", u'execution': u'<% execution() %>'}}}}, u'version': u'2.0', u'type': u'direct',         |
|               | u'name': u'send_message'}, u'set_cancel_stack_update_failed': {u'version': u'2.0', u'type': u'direct', u'name': u'set_cancel_stack_update_failed', u'publish': {u'status': u'FAILED',           |
|               | u'message': u'<% task(cancel_stack_update).result %>'}, u'on-success': u'send_message'}}, u'description': u'Cancel a currently running stack update', u'version': u'2.0', u'name':              |
|               | u'cancel_stack_update'}}}}}}']                                                                                                                                                                  |
|               |  ZaqarAction.queue_post failed: <class 'zaqarclient.transport.errors.MalformedRequest'>: Error response from Zaqar. Code: 400. Title: Invalid queue identification. Description: The format of  |
|               | the submitted queue name or project id is not valid..                                                                                                                                           |
| Created at    | 2017-02-27 13:04:46                                                                                                                                                                             |
| Updated at    | 2017-02-27 13:04:47                                                                                                                                                                             |
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


mistral task-get fa41b255-382b-44ee-bde2-31b197fb2789
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for 192.168.24.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for 192.168.24.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field         | Value                                                                                                                                                                                           |
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ID            | fa41b255-382b-44ee-bde2-31b197fb2789                                                                                                                                                            |
| Name          | cancel_stack_update                                                                                                                                                                             |
| Workflow name | tripleo.package_update.v1.cancel_stack_update                                                                                                                                                   |
| Execution ID  | d732cefd-9ade-48d2-9c39-0a5c60d092f0                                                                                                                                                            |
| State         | ERROR                                                                                                                                                                                           |
| State info    | Failed to run action [action_ex_id=09e4ea8b-54b1-4944-8e89-1db3de529512, action_cls='<class 'mistral.actions.action_factory.CancelStackUpdateAction'>', attributes='{}', params='{u'stack_id':  |
|               | u'5f54018a-938d-4503-ac21-634b9b7ff37a'}']                                                                                                                                                      |
|               |  __init__() takes exactly 5 arguments (4 given)                                                                                                                                                 |
| Created at    | 2017-02-27 13:04:46                                                                                                                                                                             |
| Updated at    | 2017-02-27 13:04:46                                                                                                                                                                             |
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Version-Release number of selected component (if applicable):
-------------------------------------------------------------
openstack-mistral-api-4.0.1-0.20170222161147.060a153.el7ost.noarch
python-openstack-mistral-4.0.1-0.20170222161147.060a153.el7ost.noarch
openstack-mistral-engine-4.0.1-0.20170222161147.060a153.el7ost.noarch
puppet-mistral-10.3.0-0.20170213130808.a6e8ebc.el7ost.noarch
python-mistralclient-3.0.0-0.20170208192040.a7bf138.el7ost.noarch
openstack-mistral-executor-4.0.1-0.20170222161147.060a153.el7ost.noarch
openstack-mistral-common-4.0.1-0.20170222161147.060a153.el7ost.noarch

Steps to Reproduce:
-------------------
1. Deploy RHOS-11 (puddle 2017-02-17.7)
2. Setup latest repos
3. Update undercloud
4. Run overcloud update
    yes '' | openstack overcloud update stack -i overcloud
5. Try to cancel update
    openstack overcloud update abort overcloud

Actual results:
---------------
Cancel operation is ignored

Additional info:
----------------
Virt setup: 3controllers + 1compute + 3ceph

Comment 5 Brad P. Crochet 2017-04-03 17:41:17 UTC
The posted patches will need to be backported to Ocata upstream.

Comment 7 Lon Hohberger 2017-10-10 18:08:51 UTC
According to our records, this should be resolved by python-tripleoclient-6.2.0-2.el7ost.  This build is available now.

Comment 10 Yurii Prokulevych 2017-12-18 13:24:22 UTC
Verified with openstack-tripleo-common-6.1.2-1.el7ost.noarch and 
python-tripleoclient-6.2.0-2.el7ost.noarch

Abort option is not available:
openstack overcloud update abort --help
Command "overcloud" matches:
  overcloud delete
  overcloud deploy
  overcloud execute
  overcloud generate fencing
  overcloud image build
  overcloud image upload
  overcloud netenv validate
  overcloud node configure
  overcloud node delete
  overcloud node import
  overcloud node introspect
  overcloud node provide
  overcloud parameters set
  overcloud plan create
  overcloud plan delete
  overcloud plan deploy
  overcloud plan list
  overcloud profiles list
  overcloud profiles match
  overcloud raid create
  overcloud update clear breakpoints
  overcloud update stack

openstack overcloud update stack --help                       
usage: openstack overcloud update stack [-h] [--templates [TEMPLATES]] [-i]
                                        [-e <HEAT ENVIRONMENT FILE>]
                                        [--answers-file ANSWERS_FILE]
                                        [stack]

Updates packages on overcloud nodes

positional arguments:
  stack                 Name or ID of heat stack to scale (default=Env:
                        OVERCLOUD_STACK_NAME)

optional arguments:
  -h, --help            show this help message and exit
  --templates [TEMPLATES]
                        The directory containing the Heat templates to deploy.
                        This argument is deprecated. The command now utilizes
                        a deployment plan, which should be updated prior to
                        running this command, should that be required.
                        Otherwise this argument will be silently ignored.
  -i, --interactive
  -e <HEAT ENVIRONMENT FILE>, --environment-file <HEAT ENVIRONMENT FILE>
                        Environment files to be passed to the heat stack-
                        create or heat stack-update command. (Can be specified
                        more than once.) This argument is deprecated. The
                        command now utilizes a deployment plan, which should
                        be updated prior to running this command, should that
                        be required. Otherwise this argument will be silently
                        ignored.
  --answers-file ANSWERS_FILE
                        Path to a YAML file with arguments and parameters.
                        DEPRECATED. Not necessary when used with a plan. Will
                        be silently ignored, and removed in the "P" release.


Note You need to log in before you can comment on or make changes to this bug.