Bug 1661810

Summary: [OVN] upgrade failed ovn osp13 to osp14 on Set fact networking_ovn_metadata_agent_enabled task
Product: Red Hat OpenStack Reporter: Eran Kuris <ekuris>
Component: openstack-tripleo-heat-templatesAssignee: Daniel Alvarez Sanchez <dalvarez>
Status: CLOSED ERRATA QA Contact: Gurenko Alex <agurenko>
Severity: urgent Docs Contact:
Priority: medium    
Version: 14.0 (Rocky)CC: amuller, apevec, augol, bcafarel, ccamacho, dalvarez, jschluet, lhh, majopela, mburns, nyechiel, scohen
Target Milestone: gaKeywords: Triaged
Target Release: 14.0 (Rocky)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-9.0.1-0.20181013060907 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-11 11:55:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
logs none

Description Eran Kuris 2018-12-23 14:37:45 UTC
Description of problem:
Processing of upgrade OVN-osp13 to osp14 failed on overcloud_upgrade_run_Compute
step in task "Set fact networking_ovn_metadata_agent_enabled"



2018-12-23 07:57:40 |  u'',
2018-12-23 07:57:40 |  u'TASK [Set fact networking_ovn_metadata_agent_enabled] **************************',
2018-12-23 07:57:40 |  u'Sunday 23 December 2018  07:57:38 -0500 (0:00:00.392)       0:01:08.108 ******* ',
2018-12-23 07:57:40 |  u'fatal: [compute-1]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: \'networking_ovn_metadata_agent_enabled\' is undefined\\n\\nThe error appears to have been in \'/var/lib/mistral/ac14c5f3-92da-450e-b216-8cebb4399cda/Compute/upgrade_tasks.yaml\': line 112, column 5, but may\\nbe elsewhere in the file depending on the exact syntax problem.\\n\\nThe offending line appears to be:\\n\\n    register: networking_ovn_metadata_agent_enabled_result\\n  - name: Set fact networking_ovn_metadata_agent_enabled\\n    ^ here\\n"}',
2018-12-23 07:57:40 |  u'fatal: [compute-0]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: \'networking_ovn_metadata_agent_enabled\' is undefined\\n\\nThe error appears to have been in \'/var/lib/mistral/ac14c5f3-92da-450e-b216-8cebb4399cda/Compute/upgrade_tasks.yaml\': line 112, column 5, but may\\nbe elsewhere in the file depending on the exact syntax problem.\\n\\nThe offending line appears to be:\\n\\n    register: networking_ovn_metadata_agent_enabled_result\\n  - name: Set fact networking_ovn_metadata_agent_enabled\\n    ^ here\\n"}',
2018-12-23 07:57:40 |  u'',
2018-12-23 07:57:40 |  u'PLAY RECAP *********************************************************************',
2018-12-23 07:57:40 |  u'compute-0                  : ok=23   changed=8    unreachable=0    failed=1   ',
2018-12-23 07:57:40 |  u'compute-1                  : ok=23   changed=8    unreachable=0    failed=1   ',
2018-12-23 07:57:40 |  u'',
2018-12-23 07:57:40 |  u'Sunday 23 December 2018  07:57:38 -0500 (0:00:00.088)       0:01:08.197 ******* ',
2018-12-23 07:57:40 |  u'=============================================================================== ']
2018-12-23 07:57:40 | [u'Failed to update nodes - Compute, please see the logs.']
@                                                                                           
Version-Release number of selected component (if applicable):

from osp13 z3 to osp14 2018-12-20.4
How reproducible:

100% 
Steps to Reproduce:
1.run upgrade job.
2.
3.

Actual results:


Expected results:


Additional info:
https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/network/view/networking-ovn/job/DFG-network-networking-ovn-upgrade-13-14_director-rhel-virthost-3cont_2comp_2net-ipv4-geneve-composable/58/

Comment 1 Eran Kuris 2018-12-24 08:40:34 UTC
Created attachment 1516483 [details]
logs

Comment 2 Daniel Alvarez Sanchez 2018-12-24 09:11:35 UTC
This patch [0] merged in stable/rocky introduced the bug which wasn't caught upstream since we don't run upgrades using ML2/OVN.
I just post a fix to review [1].


[0] https://github.com/openstack/tripleo-heat-templates/commit/56bec75c02b56bab79f35bc94c9110ee93eab82f#diff-414c32eadb8359e81046e9356fead25a
[1] https://review.openstack.org/#/c/627163

Comment 19 errata-xmlrpc 2019-01-11 11:55:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045