Bug 1502860

Summary:	rhosp-director: difficult to map certain containers to their logs.
Product:	Red Hat OpenStack	Reporter:	Alexander Chuzhoy <sasha>
Component:	openstack-tripleo-heat-templates	Assignee:	Jiri Stransky <jstransk>
Status:	CLOSED ERRATA	QA Contact:	Omri Hochman <ohochman>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	12.0 (Pike)	CC:	achernet, ahrechan, dbecker, dprince, emacchi, imain, jamsmith, jschluet, jstransk, maandre, m.andre, mburns, mcornea, morazi, ohochman, rhel-osp-director-maint, slinaber
Target Milestone:	z3	Keywords:	Triaged, ZStream
Target Release:	12.0 (Pike)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-tripleo-heat-templates-7.0.12-1.el7ost	Doc Type:	Enhancement
Doc Text:	This update helps operators locate log files after an upgrade from a non-containerized to a containerized deployment. If old log files are present when the upgrade begins, the old files are moved to a new file location. A readme.txt file is placed in the old file location. The file points to the new log file location. For example, if a /var/log/nova directory exists, a /var/log/nova/readme.txt file is created, advising the reader to look in the /var/log/containers/nova directory instead.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-08-20 12:58:30 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Alexander Chuzhoy 2017-10-16 20:45:05 UTC

rhosp-director: difficult to map certain containers to their logs.

Environment:

openstack-tripleo-heat-templates-7.0.2-0.20171007062244.el7ost.noarch
instack-undercloud-7.4.2-0.20171006021856.el7ost.noarch
openstack-puppet-modules-11.0.0-0.20170828113154.el7ost.noarch


These are the names of containers on a controller:

gnocchi_api
gnocchi_statsd
gnocchi_metricd
panko_api
nova_metadata
nova_api
glance_api
swift_account_server
aodh_listener
swift_container_auditor
heat_api_cron
swift_object_expirer
swift_object_updater
swift_container_replicator
swift_account_auditor
logrotate_crond
heat_api_cfn
nova_conductor
swift_object_replicator
swift_container_server
heat_engine
aodh_api
swift_rsync
nova_vnc_proxy
ceilometer_agent_notification
swift_account_reaper
nova_consoleauth
nova_api_cron
aodh_notifier
ceilometer_agent_central
swift_account_replicator
swift_object_auditor
heat_api
swift_proxy
swift_object_server
nova_scheduler
swift_container_updater
aodh_evaluator
keystone_cron
keystone
nova_placement
horizon
haproxy-bundle-docker-0
redis-bundle-docker-0
galera-bundle-docker-0
rabbitmq-bundle-docker-0
clustercheck
ceph-mon-overcloud-controller-0
memcached

These are the container logs:
/var/log/containers/aodh/aodh-dbsync.log
/var/log/containers/aodh/aodh-evaluator.log
/var/log/containers/aodh/aodh-notifier.log
/var/log/containers/aodh/app.log
/var/log/containers/aodh/aodh-listener.log
/var/log/containers/httpd/aodh-api/error_log
/var/log/containers/httpd/aodh-api/aodh_wsgi_error.log
/var/log/containers/httpd/aodh-api/aodh_wsgi_access.log
/var/log/containers/httpd/gnocchi-api/error_log
/var/log/containers/httpd/gnocchi-api/gnocchi_wsgi_error.log
/var/log/containers/httpd/gnocchi-api/gnocchi_wsgi_access.log
/var/log/containers/httpd/heat-api/error_log
/var/log/containers/httpd/heat-api/heat_api_wsgi_error.log
/var/log/containers/httpd/heat-api/heat_api_wsgi_access.log
/var/log/containers/httpd/heat-api-cfn/error_log
/var/log/containers/httpd/heat-api-cfn/heat_api_cfn_wsgi_error.log
/var/log/containers/httpd/heat-api-cfn/heat_api_cfn_wsgi_access.log
/var/log/containers/httpd/horizon/error_log
/var/log/containers/httpd/horizon/ssl_error_log
/var/log/containers/httpd/horizon/horizon_error.log
/var/log/containers/httpd/horizon/ssl_access_log
/var/log/containers/httpd/horizon/ssl_request_log
/var/log/containers/httpd/horizon/horizon_access.log
/var/log/containers/httpd/keystone/error_log
/var/log/containers/httpd/keystone/keystone_wsgi_main_error.log
/var/log/containers/httpd/keystone/keystone_wsgi_admin_error.log
/var/log/containers/httpd/keystone/keystone_wsgi_main_access.log
/var/log/containers/httpd/keystone/keystone_wsgi_admin_access.log
/var/log/containers/httpd/nova-api/error_log
/var/log/containers/httpd/nova-api/nova_api_wsgi_error.log
/var/log/containers/httpd/nova-api/nova_api_wsgi_access.log
/var/log/containers/httpd/nova-placement/error_log
/var/log/containers/httpd/nova-placement/placement_wsgi_error.log
/var/log/containers/httpd/nova-placement/placement_wsgi_access.log
/var/log/containers/httpd/panko-api/error_log
/var/log/containers/httpd/panko-api/panko_wsgi_error.log
/var/log/containers/httpd/panko-api/panko_wsgi_access.log
/var/log/containers/ceilometer/agent-notification.log
/var/log/containers/ceilometer/ceilometer-upgrade.log
/var/log/containers/glance/api.log
/var/log/containers/gnocchi/gnocchi-upgrade.log
/var/log/containers/gnocchi/gnocchi-metricd.log
/var/log/containers/gnocchi/gnocchi-statsd.log
/var/log/containers/gnocchi/app.log
/var/log/containers/heat/heat_api.log
/var/log/containers/heat/heat-engine.log
/var/log/containers/heat/heat_api_cfn.log
/var/log/containers/horizon/horizon.log
/var/log/containers/keystone/keystone.log
/var/log/containers/nova/nova-manage.log
/var/log/containers/nova/nova-scheduler.log
/var/log/containers/nova/nova-consoleauth.log
/var/log/containers/nova/nova-novncproxy.log
/var/log/containers/nova/nova-conductor.log
/var/log/containers/nova/nova-api-metadata.log
/var/log/containers/nova/nova-api.log
/var/log/containers/nova/nova-placement-api.log
/var/log/containers/panko/panko-dbsync.log
/var/log/containers/panko/app.log


For example looking at the above list. How does the user find the log of keystone_cron container?
There should be an easy way to correlate between a container and its log for all containers.

Comment 1 Dan Prince 2017-10-18 13:53:25 UTC

We could make use of the Ansible upgrade framework to handle this during the upgrade by doing the following for each service:

1) Move the existing logs for any baremetal services into a subdirectory. So /var/log/keystone/*.log gets moved to /var/log/keystone/backup/*.log

2) Add a README file in the /var/log/<service> directory that points to the new log file location in /var/log/containers/<service>. This will also mention that the old log files above have been moved into the 'backup' directory.

----

The above 2 things should help users who upgrade and new users know where to look for the new logs in /var/log/containers. By moving the old logs to backup directories we prevent confusion from SOS reports or other tools that might grab the old files.

We should also be more explicit about this new log location in our documentation.

Additionally, we should rename any ambiguous log files in /var/log/containers (app.log for example) to more closely match the service names.

A combination of the above should greatly help on this front.

Comment 4 Omri Hochman 2017-11-29 14:28:47 UTC

It's too late to get into osp12 GA --> should be in zstream

Comment 16 errata-xmlrpc 2018-08-20 12:58:30 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2331