Bug 1826367 - [RHOSP13]Running the openstack overcloud update run --nodes Controller command will fail if you have installed openvswitch-test rpm.
Summary: [RHOSP13]Running the openstack overcloud update run --nodes Controller comman...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 13.0 (Queens)
Hardware: All
OS: All
medium
medium
Target Milestone: z13
: 13.0 (Queens)
Assignee: Jesse Pretorius
QA Contact: Sofer Athlan-Guyot
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-04-21 14:18 UTC by MD Sufiyan
Modified: 2023-09-07 22:54 UTC (History)
11 users (show)

Fixed In Version: openstack-tripleo-heat-templates-8.4.1-63.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-28 18:23:44 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 729911 0 None MERGED Correctly match openvswitch package 2020-10-05 10:06:46 UTC
OpenStack gerrit 755358 0 None NEW [QUEENS ONLY] Fix layered upgrade workaround mechanism. 2020-10-05 10:06:46 UTC
Red Hat Issue Tracker OSP-28413 0 None None None 2023-09-07 22:54:42 UTC
Red Hat Product Errata RHBA-2020:4388 0 None None None 2020-10-28 18:24:00 UTC

Description MD Sufiyan 2020-04-21 14:18:46 UTC
Description of problem:

When running the "openstack overcloud update run --nodes Controller" then command will fail if you have installed openvswitch-test rpm.

This due to an awk in the Ansible code that needs some improvement.

The suspected awk line is in playbook Controller/update_tasks.yaml generated by the update command on line 361 whith name "Get current OpenvSwitch package name".

The respective command is:

~~~
rpm -qa | awk -F- '/^(openvswitch[0-9]+\.[0-9]+-|openvswitch-2)/{print $1}'.
~~~

This command is used to determine the current installed ovs version, but when you run this with openvswitch-test installed it will return 2 matching lines causing the ansible code to fail.

~~~
rpm -qa | awk -F- '/^(openvswitch[0-9]+\.[0-9]+-|openvswitch-2)/'
openvswitch2.11-test-2.11.0-35.el7fdp.noarch
openvswitch2.11-2.11.0-35.el7fdp.x86_64
~~~

Is their possibility to update the ansible check such that it won't fail for the returned values from the query[1] Considering "openvswitch2.11-test" is a crucial package when it comes to tcpdump on ovs interfaces which is for troubleshooting ovs-dpdk issues.

Version-Release number of selected component (if applicable):
openvswitch2.11-test-2.11.0-35.el7fdp.noarch
openvswitch2.11-2.11.0-35.el7fdp.x86_64
OSP13

How reproducible:


Steps to Reproduce:
1. deploy OSP with openvswitch
2. install openvswitch2.11-test
3. follow upgrade procedure , controller update will fail while executing "openstack overcloud update run --nodes Controller"

Actual results:
deployment failed

Expected results:
deployment should pass along with ansible check for ovs version

Comment 19 Sofer Athlan-Guyot 2020-09-30 08:41:42 UTC
Hi,

so to test this:

 1. deploy osp13z11
 2. install a matching version of openvswitch2.11-test.noarch from brew;

example from ctl-0:

Sep 29 17:46:44 Installed: openvswitch2.11-test-2.11.3-64.el7fdp.noarch

 3. run the update without failure

So this happened all right, there was no failure.  But we introduced a regression because there was no special treatment of openvswitch.

2020-09-29 14:08:00 | TASK [Get current OpenvSwitch package name] ************************************
2020-09-29 14:08:00 | Tuesday 29 September 2020  13:52:26 -0400 (0:00:00.503)       0:05:27.767 ***** 

2020-09-29 14:08:00 | changed: [controller-2] => {"changed": true, "cmd": "rpm -qa | awk -F- '/^(openvswitch[0-9]+\\.[0-9]+-[0-9]+\\.[0-9]+\\.[-0]+-|openvswitch-2)/{print $1}'", "delta": "0:00:01.258453", "end": "2020-09-29 17:52:27.655876", "rc": 0, "start": "2020-09-29    17:52:26.397423", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

We can see that the regex matched nothing.

Later during the the package update:

2020-09-29 14:08:00 | TASK [Update all packages] *****************************************************
2020-09-29 14:08:00 | Tuesday 29 September 2020  13:52:55 -0400 (0:00:00.218)       0:05:57.178 ***** 
2020-09-29 14:08:30 | changed: [controller-2] => {

---> Package resource-agents.x86_64 0:4.1.1-61.el7 will be an update
---> Package rhosp-openvswitch.noarch 0:2.11-0.6.el7ost will be updated
---> Package rhosp-openvswitch.noarch 0:2.11-0.7.el7ost will be an update
---> Package rhosp-openvswitch-ovn-central.noarch 0:2.11-0.6.el7ost will be updated
---> Package rhosp-openvswitch-ovn-central.noarch 0:2.11-0.7.el7ost will be an update
---> Package rhosp-openvswitch-ovn-host.noarch 0:2.11-0.6.el7ost will be updated
---> Package rhosp-openvswitch-ovn-host.noarch 0:2.11-0.7.el7ost will be an update
---> Package rhosp-release.noarch 0:13.0.11-1.el7ost will be updated
---> Package rhosp-release.noarch 0:13.0.13-1.el7ost will be an update
---> Package rpcbind.x86_64 0:0.2.0-48.el7 will be updated


So there is no longer a special treatment of openvswitch because the regex didn't match the currently installed packages.

In the review there is a example attached:

cat review.ex 
openvswitch2.11-test-2.11.0-35.el7fdp.noarch
openvswitch2.11-2.11.0-35.el7fdp.x86_64

and it works with that example:

cat review.ex | awk -F- '/^(openvswitch[0-9]+\.[0-9]+-[0-9]+\.[0-9]+\.[-0]+-|openvswitch-2)/{print $1}'
openvswitch2.11


Now if I take the currently update package on this osp13 example:

[root@controller-0 ~]# rpm -qa > 2020-09-16.1-full.ex
[root@controller-0 ~]# cat 2020-09-16.1-full.ex | awk -F- '/^(openvswitch[0-9]+\.[0-9]+-[0-9]+\.[0-9]+\.[-0]+-|openvswitch-2)/{print $1}'

empty, while it shouldn't:

[root@controller-0 ~]# grep '^openvswitch2' 2020-09-16.1-full.ex
openvswitch2.11-test-2.11.3-64.el7fdp.noarch
openvswitch2.11-2.11.3-64.el7fdp.x86_64

This regex works in all cases:

[root@controller-0 ~]# cat review.ex | awk -F- '/^(openvswitch[0-9]+\.[0-9]+-[0-9]+\.[0-9]+\.[0-9]+-|openvswitch-2)/{print $1}'
openvswitch2.11

[root@controller-0 ~]# cat 2020-09-16.1-full.ex | awk -F- '/^(openvswitch[0-9]+\.[0-9]+-[0-9]+\.[0-9]+\.[0-9]+-|openvswitch-2)/{print $1}'
openvswitch2.11


So, this patch introduced a regression as we don't have a special treatment of openvswitch in all cases.  I'll create a new bugzilla for it.

Moving this one to on_dev and creating a new bugzilla for the regression.

Comment 20 Sofer Athlan-Guyot 2020-09-30 17:39:30 UTC
Fix the regression in that review (main bz is https://bugzilla.redhat.com/show_bug.cgi?id=1883786), and it should be the final fix for this as well.

Now strictly speaking the issue with test doesn't happen anymore, but ... it prevent the expected workaround mechanism to be triggered.

So on one side it's Validated, on the other side without 755358, there is a regression that currently have no real world impact.

Comment 22 Jesse Pretorius 2020-10-07 10:04:20 UTC
Clearly there is a bug in this patch, but it's not a blocking issue. The intended outcome still is achieved, even if it does involve using the fallback mechanism. As such, I'll move this to verified for the z13 release and the bug in the patch will be fixed in z14: https://bugzilla.redhat.com/show_bug.cgi?id=1883786

Comment 27 errata-xmlrpc 2020-10-28 18:23:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 13.0 director bug fix advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4388


Note You need to log in before you can comment on or make changes to this bug.