Bug 1860658

Summary: Octavia Log offloading tenant flow messages with a failed variable expansion of project_id and lb_id
Product: Red Hat OpenStack Reporter: Omer Schwartz <oschwart>
Component: openstack-octaviaAssignee: Carlos Goncalves <cgoncalves>
Status: CLOSED ERRATA QA Contact: Omer Schwartz <oschwart>
Severity: medium Docs Contact:
Priority: medium    
Version: 16.1 (Train)CC: cgoncalves, ihrachys, lpeer, majopela, scohen
Target Milestone: z2Keywords: Triaged
Target Release: 16.1 (Train on RHEL 8.2)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-octavia-5.0.3-1.20200814153400.1bafec3.el8ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-10-28 15:38:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1856835, 1883536    
Bug Blocks: 1623977    

Description Omer Schwartz 2020-07-26 10:13:40 UTC
Description of problem:

Using the Octavia Log offloading feature, tenant flow messages contain the variables project_id and lb_id instead of their expansion (in the Controller, "octavia-amphora.log" file).


For example, I sent some traffic and received the next tenant flow messages:

Jul 26 05:31:07 amphora-fcf88687-dee7-47f8-a02a-62c134631bc3 haproxy[5357]: {{ project_id }} {{ lb_id }} ee7124d5-dba4-4b8e-9ba5-1015020297df 10.0.0.54 51378 26/Jul/2020:05:31:07.574 r 200 69 74 - [ssl_c_s_dn] 4f2124c3-7df6-4e03-977f-64d1b858057d:ee7124d5-dba4-4b8e-9ba5-1015020297df 804c9e28-7f6d-4e43-bbef-c02b076acea9 12 ----

We see {{ project_id }} and {{ lb_id }} instead of their expansion result.


Version-Release number of selected component (if applicable):

(overcloud) [stack@undercloud-0 ~]$ cat /var/lib/rhos-release/latest-installed
16.1  -p RHOS-16.1-RHEL-8-20200714.n.0

How reproducible:
100%, I managed to reproduce it in my environment.

Steps to Reproduce:
1. Deploy OSP 16.1 in HA
2. Set the OctaviaLogOffload flag to true in the default parameters in the /home/stack/virt/extra_templates.yaml
3. Run /home/stack/overcloud_deploy.sh
4. Send any kind of traffic
5. Enter the /var/log/containers/octavia/octavia-amphora.log file in the Controller and search a tenant flow message


Actual results:

Tenant flow message with a failed expansion of the variables:

Jul 26 05:31:07 amphora-fcf88687-dee7-47f8-a02a-62c134631bc3 haproxy[5357]: {{ project_id }} {{ lb_id }} ee7124d5-dba4-4b8e-9ba5-1015020297df 10.0.0.54 51378 26/Jul/2020:05:31:07.574 r 200 69 74 - [ssl_c_s_dn] 4f2124c3-7df6-4e03-977f-64d1b858057d:ee7124d5-dba4-4b8e-9ba5-1015020297df 804c9e28-7f6d-4e43-bbef-c02b076acea9 12 ----

In file /var/log/containers/octavia/octavia-amphora.log in the Controller.


Expected results:

Tenant flow message with "expansioned" variables:

Jul 26 05:31:07 amphora-fcf88687-dee7-47f8-a02a-62c134631bc3 haproxy[5357]: f7c23bb53ab74d05bdfeea5e0129bf10 be037678-85ce-417a-b814-0c66038c4529 ee7124d5-dba4-4b8e-9ba5-1015020297df 10.0.0.54 51378 26/Jul/2020:05:31:07.574 r 200 69 74 - [ssl_c_s_dn] 4f2124c3-7df6-4e03-977f-64d1b858057d:ee7124d5-dba4-4b8e-9ba5-1015020297df 804c9e28-7f6d-4e43-bbef-c02b076acea9 12 ----

Comment 3 Omer Schwartz 2020-09-13 14:31:27 UTC
I am seeing now that [ssl_c_s_dn] variable is also "unexpanded".

Comment 6 Omer Schwartz 2020-09-15 11:17:20 UTC
Hi,

The package openstack-octavia-5.0.3-1.20200814153400.1bafec3.el8ost is not in the passed_phase2 puddle of osp16.1.

The current passed_phase2 puddle RHOS-16.1-RHEL-8-20200903.n.0 [1] has the openstack-octavia-5.0.2-0.20200526173428.09e863e.el8ost.src.rpm package [2]



[1] http://download.eng.bos.redhat.com/rcm-guest/puddles/OpenStack/16.1-RHEL-8/RHOS-16.1-RHEL-8-20200903.n.0/

[2] http://download.eng.bos.redhat.com/rcm-guest/puddles/OpenStack/16.1-RHEL-8/passed_phase2/compose/OpenStack/source/tree/Packages/

Comment 7 Omer Schwartz 2020-09-24 14:34:20 UTC
After verification that involved these steps, I:

1. Deployed OSP 16.1 in HA
2. Changed the flag of the OctaviaLogOffload: true in /home/stack/virt/extra_templates.yaml
3. Due to bug https://bugzilla.redhat.com/show_bug.cgi?id=1856835,
I applied the following workaround: https://review.opendev.org/#/c/754079.
4. re-ran overcloud_deploy.sh
5. Created a LB and sent some traffic.
6. I saw tenant flow logs in the next format:
Sep 24 10:18:59 amphora-c8be7552-5699-48b4-9ab6-29ce52975904 haproxy[5451]: d3f6513b54b34c3bb4bf6ee58f78d030 62385ae9-3a52-41d4-87c8-4abd481e6261 726d3f53-ab58-49a8-9df5-76b30e6fe8cd 10.0.0.100 33420 24/Sep/2020:10:18:59.175 r 200 69 74 - [ssl_c_s_dn] fec80f79-2a71-4d3b-9de5-7f6bea8f5b19:726d3f53-ab58-49a8-9df5-76b30e6fe8cd 95126d1a-b72b-4099-abef-cfc00928c281 13 ----

which does not contain the "un-expantioned" variables 'project_id', and 'lb_id'.

The puddle I checked is:
(overcloud) [stack@undercloud-0 virt]$ cat /var/lib/rhos-release/latest-installed
16.1  -p RHOS-16.1-RHEL-8-20200917.n.3

Looks good to me.

Comment 12 errata-xmlrpc 2020-10-28 15:38:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:4284