Bug 1860658 - Octavia Log offloading tenant flow messages with a failed variable expansion of project_id and lb_id
Summary: Octavia Log offloading tenant flow messages with a failed variable expansion ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-octavia
Version: 16.1 (Train)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: z2
: 16.1 (Train on RHEL 8.2)
Assignee: Carlos Goncalves
QA Contact: Omer Schwartz
URL:
Whiteboard:
Depends On: 1856835 1883536
Blocks: 1623977
TreeView+ depends on / blocked
 
Reported: 2020-07-26 10:13 UTC by Omer Schwartz
Modified: 2022-08-23 18:30 UTC (History)
5 users (show)

Fixed In Version: openstack-octavia-5.0.3-1.20200814153400.1bafec3.el8ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-28 15:38:39 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack Storyboard 2007958 0 None None None 2020-07-26 15:01:37 UTC
OpenStack gerrit 743059 0 None MERGED Fix haproxy user flow log format substitution 2020-12-08 13:05:07 UTC
OpenStack gerrit 743724 0 None MERGED Fix haproxy user flow log format substitution 2020-12-08 13:05:06 UTC
Red Hat Issue Tracker OSP-3268 0 None None None 2022-08-23 18:30:03 UTC
Red Hat Product Errata RHEA-2020:4284 0 None None None 2020-10-28 15:39:22 UTC

Description Omer Schwartz 2020-07-26 10:13:40 UTC
Description of problem:

Using the Octavia Log offloading feature, tenant flow messages contain the variables project_id and lb_id instead of their expansion (in the Controller, "octavia-amphora.log" file).


For example, I sent some traffic and received the next tenant flow messages:

Jul 26 05:31:07 amphora-fcf88687-dee7-47f8-a02a-62c134631bc3 haproxy[5357]: {{ project_id }} {{ lb_id }} ee7124d5-dba4-4b8e-9ba5-1015020297df 10.0.0.54 51378 26/Jul/2020:05:31:07.574 r 200 69 74 - [ssl_c_s_dn] 4f2124c3-7df6-4e03-977f-64d1b858057d:ee7124d5-dba4-4b8e-9ba5-1015020297df 804c9e28-7f6d-4e43-bbef-c02b076acea9 12 ----

We see {{ project_id }} and {{ lb_id }} instead of their expansion result.


Version-Release number of selected component (if applicable):

(overcloud) [stack@undercloud-0 ~]$ cat /var/lib/rhos-release/latest-installed
16.1  -p RHOS-16.1-RHEL-8-20200714.n.0

How reproducible:
100%, I managed to reproduce it in my environment.

Steps to Reproduce:
1. Deploy OSP 16.1 in HA
2. Set the OctaviaLogOffload flag to true in the default parameters in the /home/stack/virt/extra_templates.yaml
3. Run /home/stack/overcloud_deploy.sh
4. Send any kind of traffic
5. Enter the /var/log/containers/octavia/octavia-amphora.log file in the Controller and search a tenant flow message


Actual results:

Tenant flow message with a failed expansion of the variables:

Jul 26 05:31:07 amphora-fcf88687-dee7-47f8-a02a-62c134631bc3 haproxy[5357]: {{ project_id }} {{ lb_id }} ee7124d5-dba4-4b8e-9ba5-1015020297df 10.0.0.54 51378 26/Jul/2020:05:31:07.574 r 200 69 74 - [ssl_c_s_dn] 4f2124c3-7df6-4e03-977f-64d1b858057d:ee7124d5-dba4-4b8e-9ba5-1015020297df 804c9e28-7f6d-4e43-bbef-c02b076acea9 12 ----

In file /var/log/containers/octavia/octavia-amphora.log in the Controller.


Expected results:

Tenant flow message with "expansioned" variables:

Jul 26 05:31:07 amphora-fcf88687-dee7-47f8-a02a-62c134631bc3 haproxy[5357]: f7c23bb53ab74d05bdfeea5e0129bf10 be037678-85ce-417a-b814-0c66038c4529 ee7124d5-dba4-4b8e-9ba5-1015020297df 10.0.0.54 51378 26/Jul/2020:05:31:07.574 r 200 69 74 - [ssl_c_s_dn] 4f2124c3-7df6-4e03-977f-64d1b858057d:ee7124d5-dba4-4b8e-9ba5-1015020297df 804c9e28-7f6d-4e43-bbef-c02b076acea9 12 ----

Comment 3 Omer Schwartz 2020-09-13 14:31:27 UTC
I am seeing now that [ssl_c_s_dn] variable is also "unexpanded".

Comment 6 Omer Schwartz 2020-09-15 11:17:20 UTC
Hi,

The package openstack-octavia-5.0.3-1.20200814153400.1bafec3.el8ost is not in the passed_phase2 puddle of osp16.1.

The current passed_phase2 puddle RHOS-16.1-RHEL-8-20200903.n.0 [1] has the openstack-octavia-5.0.2-0.20200526173428.09e863e.el8ost.src.rpm package [2]



[1] http://download.eng.bos.redhat.com/rcm-guest/puddles/OpenStack/16.1-RHEL-8/RHOS-16.1-RHEL-8-20200903.n.0/

[2] http://download.eng.bos.redhat.com/rcm-guest/puddles/OpenStack/16.1-RHEL-8/passed_phase2/compose/OpenStack/source/tree/Packages/

Comment 7 Omer Schwartz 2020-09-24 14:34:20 UTC
After verification that involved these steps, I:

1. Deployed OSP 16.1 in HA
2. Changed the flag of the OctaviaLogOffload: true in /home/stack/virt/extra_templates.yaml
3. Due to bug https://bugzilla.redhat.com/show_bug.cgi?id=1856835,
I applied the following workaround: https://review.opendev.org/#/c/754079.
4. re-ran overcloud_deploy.sh
5. Created a LB and sent some traffic.
6. I saw tenant flow logs in the next format:
Sep 24 10:18:59 amphora-c8be7552-5699-48b4-9ab6-29ce52975904 haproxy[5451]: d3f6513b54b34c3bb4bf6ee58f78d030 62385ae9-3a52-41d4-87c8-4abd481e6261 726d3f53-ab58-49a8-9df5-76b30e6fe8cd 10.0.0.100 33420 24/Sep/2020:10:18:59.175 r 200 69 74 - [ssl_c_s_dn] fec80f79-2a71-4d3b-9de5-7f6bea8f5b19:726d3f53-ab58-49a8-9df5-76b30e6fe8cd 95126d1a-b72b-4099-abef-cfc00928c281 13 ----

which does not contain the "un-expantioned" variables 'project_id', and 'lb_id'.

The puddle I checked is:
(overcloud) [stack@undercloud-0 virt]$ cat /var/lib/rhos-release/latest-installed
16.1  -p RHOS-16.1-RHEL-8-20200917.n.3

Looks good to me.

Comment 12 errata-xmlrpc 2020-10-28 15:38:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:4284


Note You need to log in before you can comment on or make changes to this bug.