Bug 1883954 - Amphora tenant flow log messages join administrative log file instead of having their own file
Summary: Amphora tenant flow log messages join administrative log file instead of havi...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-octavia
Version: 16.1 (Train)
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: z2
: ---
Assignee: Michael Johnson
QA Contact: Bruna Bonguardo
Greg Rakauskas
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-09-30 15:05 UTC by Michael Johnson
Modified: 2020-10-07 15:06 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
When you enable Load-balancing service instance (amphora) log offloading, both the administrative logs and the tenant logs are written to the same file (`octavia-amphora.log`). This is a known issue caused by an incorrect default value for the Orchestration service (heat) parameter, `OctaviaTenantLogFacility`. Workaround: in a custom environment file, set `OctaviaTenantLogFacility` to zero (0) and run the `openstack overcloud deploy` command: ---- parameter_defaults: OctaviaLogOffload: true OctaviaTenantLogFacility: 0 ... ---- For more information, see link:https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/director_installation_and_usage/performing-basic-overcloud-administration-tasks#modifying-the-overcloud-environment[Modifying the overcloud environment]
Clone Of:
Environment:
Last Closed: 2020-10-07 15:06:38 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 754224 0 None MERGED Fix Octavia OctaviaTenantLogFacility default 2020-11-04 23:12:30 UTC
Red Hat Bugzilla 1881362 0 medium CLOSED [Octavia][16.1] Amphora tenant flow log messages join administrative log file instead of having their own file 2021-09-15 07:09:47 UTC

Internal Links: 1881362

Description Michael Johnson 2020-09-30 15:05:25 UTC
This bug was initially created as a copy of Bug #1881362

I am copying this bug because: 
Creating a release note for 16.1.z2.


Description of problem:
Using the Octavia Log offloading feature, tenant flow log messages join the administrative log messages at 'octavia-amphora.log' file instead of having their own file named 'octavia-tenant-traffic.log'.

For example, I sent some traffic and if I am seeing the last log message of 'octavia-amphora.log' file, I see a tenant flow message:

[root@controller-0 octavia]# tail -n 1 octavia-amphora.log
Sep 22 05:13:37 amphora-ce6df23f-5c40-40ec-b20f-3217a581189c haproxy[5315]: {{ project_id }} {{ lb_id }} f9878496-ef26-40a2-9827-bfe304bd8af2 10.0.0.29 33218 22/Sep/2020:05:13:37.085 r 200 69 74 - [ssl_c_s_dn] 5213aaaa-b2a8-4edb-b397-d55544f3adbf:f9878496-ef26-40a2-9827-bfe304bd8af2 8bc46576-d6e7-44d1-a45d-2c64b9495448 2 ----
[root@controller-0 octavia]#

There is no 'octavia-tenant-traffic.log' file:

[root@controller-0 octavia]# ls -la | grep tenant
[root@controller-0 octavia]#

Version-Release number of selected component (if applicable):
(overcloud) [stack@undercloud-0 ~]$ cat /var/lib/rhos-release/latest-installed
16.1  -p RHOS-16.1-RHEL-8-20200813.n.0

How reproducible:
100%, I managed to reproduce it in my environment.

Steps to Reproduce:
1. Deploy OSP 16.1 in HA
2. Change the flag of the OctaviaLogOffload: true in /home/stack/virt/extra_templates.yaml
3. Due to bug https://bugzilla.redhat.com/show_bug.cgi?id=1856835, copy the templates folder in the following way:
sudo cp -r /usr/share/ansible/roles/octavia_controller_post_config/templates /usr/share/ansible/roles/octavia-controller-post-config/templates
4. run overcloud_deploy.sh
5. Create a LB and send some traffic.
6. Check if there is any 'octavia-tenant-traffic.log' file in the controller.


Actual results:
There is no 'octavia-tenant-traffic.log' file in the controller, but the tenant flow log messages joined the 'octavia-amphora.log' file.

Expected results:
Tenant flow logs should be in 'octavia-tenant-traffic.log' file.

Comment 6 Omer Schwartz 2020-10-01 11:03:42 UTC
Hey Greg and Michael,

I ran the overcloud deploy with the arguments Michael provided (OctaviaLogOffload:true, OctaviaTenantLogFacility:0) and the bug is still occurring (I still send traffic and receive the offloaded logs in one file, the 'octavia-amphora.log').

BTW, I tried to provide the arguments as they are and the Ansible complained that a space is missing there (should be OctaviaLogOffload: true and OctaviaTenantLogFacility: 0), so I would suggest to add the spaces to the docs.

Am I doing this in a wrong way?


Thanks

Comment 7 Omer Schwartz 2020-10-01 12:34:50 UTC
I missed Michael's email which said that I should try the workaround on new LBs instead of existing ones. Sorry for that.

It works.

Greg - can you add the spaces (' ') in the docs? (i.e. 'OctaviaLogOffload: true' instead of 'OctaviaLogOffload:true' etc).

If I run the command as the arguments are written, I receive the following error:

Processing templates in the directory /tmp/tripleoclient-9ad0seo8/tripleo-heat-templates
Exception occured while running the command
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1004, in take_action
    self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 388, in _deploy_tripleo_heat_templates_tmpdir
    new_tht_root, tht_root)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 464, in _deploy_tripleo_heat_templates
    cleanup=(not parsed_args.no_cleanup))
  File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 1341, in process_multiple_environments
    localenv = template_utils.deep_update(localenv, env)
  File "/usr/lib/python3.6/site-packages/heatclient/common/template_utils.py", line 201, in deep_update
    r = deep_update(old.get(k, {}), v)
  File "/usr/lib/python3.6/site-packages/heatclient/common/template_utils.py", line 207, in deep_update
    old[k] = new[k]
TypeError: 'str' object does not support item assignment
'str' object does not support item assignment


Besides that issue, I will gladly change the BZ state to VERIFIED.

Thanks!

Comment 8 Michael Johnson 2020-10-01 14:06:14 UTC
Glad you got it going.

Comment 10 Omer Schwartz 2020-10-01 16:15:39 UTC
Great Greg, looks good to me, thanks!

Comment 11 Carlos Goncalves 2020-10-07 15:06:38 UTC
This is not a code bug. The known issue will be documented in the 16.1.2 release notes. See https://bugzilla.redhat.com/show_bug.cgi?id=1623977


Note You need to log in before you can comment on or make changes to this bug.