Bug 1792890

Summary:	[OSP13] Manila upgrade tasks failing with: The conditional check 'manila_share_pcs_res\|succeeded' failed.
Product:	Red Hat OpenStack	Reporter:	Jose Luis Franco <jfrancoa>
Component:	openstack-tripleo-heat-templates	Assignee:	Jose Luis Franco <jfrancoa>
Status:	CLOSED ERRATA	QA Contact:	Ronnie Rasouli <rrasouli>
Severity:	high	Docs Contact:
Priority:	high
Version:	13.0 (Queens)	CC:	gouthamr, lbezdick, mburns, tbarron
Target Milestone:	---	Keywords:	Triaged, ZStream
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-tripleo-heat-templates-8.4.1-35.el7ost.src.rpm	Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:
Clones:	1792901 (view as bug list)		Environment:
Last Closed:	2020-03-10 11:23:27 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1792901

Description Jose Luis Franco 2020-01-20 10:33:40 UTC

Description of problem:

Bugzilla opened from an issue identified in the case https://access.redhat.com/support/cases/#/case/02406707.

During the upgrade of the controllers from OSP12 to OSP13, one of the upgrade tasks failed with the following error:

2020-01-18 01:59:07,802 p=136626 u=mistral |  TASK [Disable the Manila-Share cluster resource before container upgrade] ******
2020-01-18 01:59:07,802 p=136626 u=mistral |  Saturday 18 January 2020  01:59:07 +0000 (0:00:00.372)       0:05:48.717 ******
2020-01-18 01:59:07,895 p=136626 u=mistral |  skipping: [fws-shk01-fcn03] => {"changed": false, "skip_reason": "Conditional result was False"}
2020-01-18 01:59:07,991 p=136626 u=mistral |  skipping: [fws-shk01-fcn02] => {"changed": false, "skip_reason": "Conditional result was False"}
2020-01-18 01:59:08,005 p=136626 u=mistral |  [DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using
`result|succeeded` use `result is succeeded`. This feature will be removed in
version 2.9. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
2020-01-18 01:59:08,009 p=136626 u=mistral |  fatal: [fws-shk01-fcn01]: FAILED! => {"msg": "The conditional check 'manila_share_pcs_res|succeeded' failed. The error was: The failed test expects a dictionary\n\nThe error appears to have been in '/var/lib/mistral/527eac46-145c-4244-a399-e8f20cb697e8/Controller/upgrade_tasks.yaml': line 1010, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n- block:\n  - name: Disable the Manila-Share cluster resource before container upgrade\n    ^ here\n"}
2020-01-18 01:59:08,096 p=136626 u=mistral |  TASK [Update the Manila-Share bundle to use the new container image name] ******
2020-01-18 01:59:08,096 p=136626 u=mistral |  Saturday 18 January 2020  01:59:08 +0000 (0:00:00.294)       0:05:49.011 ******


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Deploy an OSP12 environment with the manilla service enabled
2. Upgrade the Undercloud to 13
3. Upgrade the Overcloud Controller's role

Actual results:

The upgrade procedure fails for the Controller.

Expected results:


Additional info:

Comment 1 Jose Luis Franco 2020-01-20 10:38:37 UTC

The upgrade tasks for the manila-share service are incorrect when evaluating the manila_share_pcs_res variable:

        - name: Update openstack-manila-share pcs resource bundle for new container image
          when:
            - step|int == 1
            - manila_share_containerized|bool
            - is_bootstrap_node
            - manila_share_pcs_res|succeeded
          block:
            - name: Disable the Manila-Share cluster resource before container upgrade
              pacemaker_resource:
                resource: openstack-manila-share
                state: disable
                wait_for_resource: true
              register: output
              retries: 5
              until: output.rc == 0

https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/docker/services/pacemaker/manila-share.yaml#L330

The manila_share_pcs_res is a boolean:

2020-01-18 01:54:07,460 p=136626 u=mistral |  TASK [Set fact manila_share_pcs_res] *******************************************
2020-01-18 01:54:07,460 p=136626 u=mistral |  Saturday 18 January 2020  01:54:07 +0000 (0:00:01.193)       0:00:48.375 ******
2020-01-18 01:54:07,477 p=136626 u=mistral |  [DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using
`result|succeeded` use `result is succeeded`. This feature will be removed in
version 2.9. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
2020-01-18 01:54:07,556 p=136626 u=mistral |  ok: [fws-shk01-fcn03] => {"ansible_facts": {"manila_share_pcs_res": true}, "changed": false}
2020-01-18 01:54:07,565 p=136626 u=mistral |  [DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using
`result|succeeded` use `result is succeeded`. This feature will be removed in
version 2.9. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
2020-01-18 01:54:07,650 p=136626 u=mistral |  ok: [fws-shk01-fcn02] => {"ansible_facts": {"manila_share_pcs_res": true}, "changed": false}
2020-01-18 01:54:07,662 p=136626 u=mistral |  [DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using
`result|succeeded` use `result is succeeded`. This feature will be removed in
version 2.9. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
2020-01-18 01:54:07,686 p=136626 u=mistral |  ok: [fws-shk01-fcn01] => {"ansible_facts": {"manila_share_pcs_res": true}, "changed": false}


Therefore, we should be using the |bool filter instead.
The same issue is occuring in the ovn-dbs.yaml template: https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/docker/services/pacemaker/ovn-dbs.yaml#L353

Comment 9 errata-xmlrpc 2020-03-10 11:23:27 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0760