Bug 1883954

Summary:	Amphora tenant flow log messages join administrative log file instead of having their own file
Product:	Red Hat OpenStack	Reporter:	Michael Johnson <michjohn>
Component:	openstack-octavia	Assignee:	Michael Johnson <michjohn>
Status:	CLOSED NOTABUG	QA Contact:	Bruna Bonguardo <bbonguar>
Severity:	medium	Docs Contact:	Greg Rakauskas <gregraka>
Priority:	unspecified
Version:	16.1 (Train)	CC:	cgoncalves, gregraka, ihrachys, lpeer, majopela, oschwart, scohen
Target Milestone:	z2	Keywords:	ReleaseNotes
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Known Issue
Doc Text:	When you enable Load-balancing service instance (amphora) log offloading, both the administrative logs and the tenant logs are written to the same file (`octavia-amphora.log`). This is a known issue caused by an incorrect default value for the Orchestration service (heat) parameter, `OctaviaTenantLogFacility`. Workaround: in a custom environment file, set `OctaviaTenantLogFacility` to zero (0) and run the `openstack overcloud deploy` command: ---- parameter_defaults: OctaviaLogOffload: true OctaviaTenantLogFacility: 0 ... ---- For more information, see link:https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/director_installation_and_usage/performing-basic-overcloud-administration-tasks#modifying-the-overcloud-environment[Modifying the overcloud environment]	Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-10-07 15:06:38 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Michael Johnson 2020-09-30 15:05:25 UTC

This bug was initially created as a copy of Bug #1881362

I am copying this bug because: 
Creating a release note for 16.1.z2.


Description of problem:
Using the Octavia Log offloading feature, tenant flow log messages join the administrative log messages at 'octavia-amphora.log' file instead of having their own file named 'octavia-tenant-traffic.log'.

For example, I sent some traffic and if I am seeing the last log message of 'octavia-amphora.log' file, I see a tenant flow message:

[root@controller-0 octavia]# tail -n 1 octavia-amphora.log
Sep 22 05:13:37 amphora-ce6df23f-5c40-40ec-b20f-3217a581189c haproxy[5315]: {{ project_id }} {{ lb_id }} f9878496-ef26-40a2-9827-bfe304bd8af2 10.0.0.29 33218 22/Sep/2020:05:13:37.085 r 200 69 74 - [ssl_c_s_dn] 5213aaaa-b2a8-4edb-b397-d55544f3adbf:f9878496-ef26-40a2-9827-bfe304bd8af2 8bc46576-d6e7-44d1-a45d-2c64b9495448 2 ----
[root@controller-0 octavia]#

There is no 'octavia-tenant-traffic.log' file:

[root@controller-0 octavia]# ls -la | grep tenant
[root@controller-0 octavia]#

Version-Release number of selected component (if applicable):
(overcloud) [stack@undercloud-0 ~]$ cat /var/lib/rhos-release/latest-installed
16.1  -p RHOS-16.1-RHEL-8-20200813.n.0

How reproducible:
100%, I managed to reproduce it in my environment.

Steps to Reproduce:
1. Deploy OSP 16.1 in HA
2. Change the flag of the OctaviaLogOffload: true in /home/stack/virt/extra_templates.yaml
3. Due to bug https://bugzilla.redhat.com/show_bug.cgi?id=1856835, copy the templates folder in the following way:
sudo cp -r /usr/share/ansible/roles/octavia_controller_post_config/templates /usr/share/ansible/roles/octavia-controller-post-config/templates
4. run overcloud_deploy.sh
5. Create a LB and send some traffic.
6. Check if there is any 'octavia-tenant-traffic.log' file in the controller.


Actual results:
There is no 'octavia-tenant-traffic.log' file in the controller, but the tenant flow log messages joined the 'octavia-amphora.log' file.

Expected results:
Tenant flow logs should be in 'octavia-tenant-traffic.log' file.

Comment 6 Omer Schwartz 2020-10-01 11:03:42 UTC

Hey Greg and Michael,

I ran the overcloud deploy with the arguments Michael provided (OctaviaLogOffload:true, OctaviaTenantLogFacility:0) and the bug is still occurring (I still send traffic and receive the offloaded logs in one file, the 'octavia-amphora.log').

BTW, I tried to provide the arguments as they are and the Ansible complained that a space is missing there (should be OctaviaLogOffload: true and OctaviaTenantLogFacility: 0), so I would suggest to add the spaces to the docs.

Am I doing this in a wrong way?


Thanks

Comment 7 Omer Schwartz 2020-10-01 12:34:50 UTC

I missed Michael's email which said that I should try the workaround on new LBs instead of existing ones. Sorry for that.

It works.

Greg - can you add the spaces (' ') in the docs? (i.e. 'OctaviaLogOffload: true' instead of 'OctaviaLogOffload:true' etc).

If I run the command as the arguments are written, I receive the following error:

Processing templates in the directory /tmp/tripleoclient-9ad0seo8/tripleo-heat-templates
Exception occured while running the command
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1004, in take_action
    self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 388, in _deploy_tripleo_heat_templates_tmpdir
    new_tht_root, tht_root)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 464, in _deploy_tripleo_heat_templates
    cleanup=(not parsed_args.no_cleanup))
  File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 1341, in process_multiple_environments
    localenv = template_utils.deep_update(localenv, env)
  File "/usr/lib/python3.6/site-packages/heatclient/common/template_utils.py", line 201, in deep_update
    r = deep_update(old.get(k, {}), v)
  File "/usr/lib/python3.6/site-packages/heatclient/common/template_utils.py", line 207, in deep_update
    old[k] = new[k]
TypeError: 'str' object does not support item assignment
'str' object does not support item assignment


Besides that issue, I will gladly change the BZ state to VERIFIED.

Thanks!

Comment 8 Michael Johnson 2020-10-01 14:06:14 UTC

Glad you got it going.

Comment 10 Omer Schwartz 2020-10-01 16:15:39 UTC

Great Greg, looks good to me, thanks!

Comment 11 Carlos Goncalves 2020-10-07 15:06:38 UTC

This is not a code bug. The known issue will be documented in the 16.1.2 release notes. See https://bugzilla.redhat.com/show_bug.cgi?id=1623977