Bug 1889654 - The ovn-dbs update task fails on bootstrap node 'ovn_dbs_image_latest' is undefined.
Summary: The ovn-dbs update task fails on bootstrap node 'ovn_dbs_image_latest' is und...
Keywords:
Status: VERIFIED
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 16.2 (Train)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: beta
: 16.2 (Train on RHEL 8.4)
Assignee: mbollo
QA Contact: Ronnie Rasouli
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-10-20 10:25 UTC by Ronnie Rasouli
Modified: 2020-11-25 15:12 UTC (History)
3 users (show)

Fixed In Version: openstack-tripleo-heat-templates-11.3.2-1.20200914170158.el8ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
OpenStack gerrit 747515 None MERGED Fix up ovn-dbs update tasks 2020-11-25 15:05:36 UTC

Description Ronnie Rasouli 2020-10-20 10:25:56 UTC
This bug was initially created as a copy of Bug #1881859

I am copying this bug because: 

Similar issue found on RHOS16.2 
from:
core_puddle: RHOS-16.2-RHEL-8-20201001.n.1
to
core_puddle: RHOS-16.2-RHEL-8-20201001.n.

I was testing a minor update on master and noticed the following error on the ovn-dbs bootstrap node (controller-0):


2020-10-19 11:22:36,028 p=925 u=mistral n=ansible | changed: [controller-0] => {"changed": true, "cmd": "pcs resource cleanup ovn-dbs-bundle", "delta": "0:00:00.956353", "end": "2020-10-19 11:22:35.946714", "rc": 0, "start": "2020-10-19 11:22:34.990361", "stderr": "", "stderr_lines": [], "stdout": "Cleaned up ovn-dbs-bundle-podman-0 on controller-2\nCleaned up ovn-dbs-bundle-podman-0 on controller-1\nCleaned up ovn-dbs-bundle-podman-0 on controller-0\nCleaned up ovn-dbs-bundle-0 on controller-2\nCleaned up ovn-dbs-bundle-0 on controller-1\nCleaned up ovn-dbs-bundle-podman-1 on controller-2\nCleaned up ovn-dbs-bundle-podman-1 on controller-1\nCleaned up ovn-dbs-bundle-podman-1 on controller-0\nCleaned up ovn-dbs-bundle-1 on controller-2\nCleaned up ovn-dbs-bundle-1 on controller-1\nCleaned up ovn-dbs-bundle-1 on controller-0\nCleaned up ovn-dbs-bundle-podman-2 on controller-2\nCleaned up ovn-dbs-bundle-podman-2 on controller-1\nCleaned up ovn-dbs-bundle-podman-2 on controller-0\nCleaned up ovn-dbs-bundle-2 on controller-2\nCleaned up ovn-dbs-bundle-2 on controller-1\nCleaned up ovn-dbs-bundle-2 on controller-0\nCleaned up ovndb_servers:1 on ovn-dbs-bundle-1\nCleaned up ovndb_servers:2 on ovn-dbs-bundle-2", "stdout_lines": ["Cleaned up ovn-dbs-bundle-podman-0 on controller-2", "Cleaned up ovn-dbs-bundle-podman-0 on controller-1", "Cleaned up ovn-dbs-bundle-podman-0 on controller-0", "Cleaned up ovn-dbs-bundle-0 on controller-2", "Cleaned up ovn-dbs-bundle-0 on controller-1", "Cleaned up ovn-dbs-bundle-podman-1 on controller-2", "Cleaned up ovn-dbs-bundle-podman-1 on controller-1", "Cleaned up ovn-dbs-bundle-podman-1 on controller-0", "Cleaned up ovn-dbs-bundle-1 on controller-2", "Cleaned up ovn-dbs-bundle-1 on controller-1", "Cleaned up ovn-dbs-bundle-1 on controller-0", "Cleaned up ovn-dbs-bundle-podman-2 on controller-2", "Cleaned up ovn-dbs-bundle-podman-2 on controller-1", "Cleaned up ovn-dbs-bundle-podman-2 on controller-0", "Cleaned up ovn-dbs-bundle-2 on controller-2", "Cleaned up ovn-dbs-bundle-2 on controller-1", "Cleaned up ovn-dbs-bundle-2 on controller-0", "Cleaned up ovndb_servers:1 on ovn-dbs-bundle-1", "Cleaned up ovndb_servers:2 on ovn-dbs-bundle-2"]}
2020-10-19 11:22:36,099 p=925 u=mistral n=ansible | TASK [Remove the ban] **********************************************************
2020-10-19 11:22:36,099 p=925 u=mistral n=ansible | Monday 19 October 2020  11:22:36 +0000 (0:00:01.400)       0:02:49.030 ******** 
2020-10-19 11:22:37,781 p=925 u=mistral n=ansible | changed: [controller-0] => {"changed": true, "cmd": "pcs resource clear ovn-dbs-bundle", "delta": "0:00:01.315285", "end": "2020-10-19 11:22:37.726092", "rc": 0, "start": "2020-10-19 11:22:36.410807", "stderr": "", "stderr_lines": [], "stdout": "Removing constraint: cli-ban-ovn-dbs-bundle-on-controller-0", "stdout_lines": ["Removing constraint: cli-ban-ovn-dbs-bundle-on-controller-0"]}
2020-10-19 11:22:37,861 p=925 u=mistral n=ansible | TASK [Get the present image used by ovn-dbs-bundle] ****************************
2020-10-19 11:22:37,861 p=925 u=mistral n=ansible | Monday 19 October 2020  11:22:37 +0000 (0:00:01.761)       0:02:50.791 ******** 
2020-10-19 11:22:39,246 p=925 u=mistral n=ansible | changed: [controller-0] => {"changed": true, "cmd": "pcs resource config ovn-dbs-bundle | grep -Eo 'image=[^ ]+' | awk -F= '{print $2;}'", "delta": "0:00:00.996193", "end": "2020-10-19 11:22:39.148661", "rc": 0, "start": "2020-10-19 11:22:38.152468", "stderr": "", "stderr_lines": [], "stdout": "cluster.common.tag/rhosp16-openstack-ovn-northd:pcmklatest", "stdout_lines": ["cluster.common.tag/rhosp16-openstack-ovn-northd:pcmklatest"]}
2020-10-19 11:22:39,317 p=925 u=mistral n=ansible | TASK [Update the ovn-dbs-bundle to use the new container image name] ***********
2020-10-19 11:22:39,318 p=925 u=mistral n=ansible | Monday 19 October 2020  11:22:39 +0000 (0:00:01.456)       0:02:52.248 ******** 
2020-10-19 11:22:39,378 p=925 u=mistral n=ansible | fatal: [controller-0]: FAILED! => {"msg": "The conditional check 'ovn_dbs_current_image.stdout != ovn_dbs_image_latest' failed. The error was: error while evaluating conditional (ovn_dbs_current_image.stdout != ovn_dbs_image_latest): 'ovn_dbs_image_latest' is undefined\n\nThe error appears to be in '/var/lib/mistral/4328790a-8e06-49c6-ab4c-1471449826a6/Controller/update_tasks.yaml': line 188, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n  - block:\n    - command: pcs resource bundle update ovn-dbs-bundle container image={{ovn_dbs_image_latest}}\n      ^ here\nWe could be wrong, but this one looks like it might be an issue with\nmissing quotes. Always quote template expression brackets when they\nstart a value. For instance:\n\n    with_items:\n      - {{ foo }}\n\nShould be written as:\n\n    with_items:\n      - \"{{ foo }}\"\n"}


Note You need to log in before you can comment on or make changes to this bug.