Bug 1502860 - rhosp-director: difficult to map certain containers to their logs.
Summary: rhosp-director: difficult to map certain containers to their logs.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 12.0 (Pike)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: z3
: 12.0 (Pike)
Assignee: Jiri Stransky
QA Contact: Omri Hochman
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-10-16 20:45 UTC by Alexander Chuzhoy
Modified: 2018-08-20 12:59 UTC (History)
17 users (show)

Fixed In Version: openstack-tripleo-heat-templates-7.0.12-1.el7ost
Doc Type: Enhancement
Doc Text:
This update helps operators locate log files after an upgrade from a non-containerized to a containerized deployment. If old log files are present when the upgrade begins, the old files are moved to a new file location. A readme.txt file is placed in the old file location. The file points to the new log file location. For example, if a /var/log/nova directory exists, a /var/log/nova/readme.txt file is created, advising the reader to look in the /var/log/containers/nova directory instead.
Clone Of:
Environment:
Last Closed: 2018-08-20 12:58:30 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:2331 None None None 2018-08-20 12:59:12 UTC
OpenStack gerrit 520570 None None None 2017-11-21 14:15:07 UTC
Launchpad 1730957 None None None 2017-11-08 14:13:43 UTC

Description Alexander Chuzhoy 2017-10-16 20:45:05 UTC
rhosp-director: difficult to map certain containers to their logs.

Environment:

openstack-tripleo-heat-templates-7.0.2-0.20171007062244.el7ost.noarch
instack-undercloud-7.4.2-0.20171006021856.el7ost.noarch
openstack-puppet-modules-11.0.0-0.20170828113154.el7ost.noarch


These are the names of containers on a controller:

gnocchi_api
gnocchi_statsd
gnocchi_metricd
panko_api
nova_metadata
nova_api
glance_api
swift_account_server
aodh_listener
swift_container_auditor
heat_api_cron
swift_object_expirer
swift_object_updater
swift_container_replicator
swift_account_auditor
logrotate_crond
heat_api_cfn
nova_conductor
swift_object_replicator
swift_container_server
heat_engine
aodh_api
swift_rsync
nova_vnc_proxy
ceilometer_agent_notification
swift_account_reaper
nova_consoleauth
nova_api_cron
aodh_notifier
ceilometer_agent_central
swift_account_replicator
swift_object_auditor
heat_api
swift_proxy
swift_object_server
nova_scheduler
swift_container_updater
aodh_evaluator
keystone_cron
keystone
nova_placement
horizon
haproxy-bundle-docker-0
redis-bundle-docker-0
galera-bundle-docker-0
rabbitmq-bundle-docker-0
clustercheck
ceph-mon-overcloud-controller-0
memcached

These are the container logs:
/var/log/containers/aodh/aodh-dbsync.log
/var/log/containers/aodh/aodh-evaluator.log
/var/log/containers/aodh/aodh-notifier.log
/var/log/containers/aodh/app.log
/var/log/containers/aodh/aodh-listener.log
/var/log/containers/httpd/aodh-api/error_log
/var/log/containers/httpd/aodh-api/aodh_wsgi_error.log
/var/log/containers/httpd/aodh-api/aodh_wsgi_access.log
/var/log/containers/httpd/gnocchi-api/error_log
/var/log/containers/httpd/gnocchi-api/gnocchi_wsgi_error.log
/var/log/containers/httpd/gnocchi-api/gnocchi_wsgi_access.log
/var/log/containers/httpd/heat-api/error_log
/var/log/containers/httpd/heat-api/heat_api_wsgi_error.log
/var/log/containers/httpd/heat-api/heat_api_wsgi_access.log
/var/log/containers/httpd/heat-api-cfn/error_log
/var/log/containers/httpd/heat-api-cfn/heat_api_cfn_wsgi_error.log
/var/log/containers/httpd/heat-api-cfn/heat_api_cfn_wsgi_access.log
/var/log/containers/httpd/horizon/error_log
/var/log/containers/httpd/horizon/ssl_error_log
/var/log/containers/httpd/horizon/horizon_error.log
/var/log/containers/httpd/horizon/ssl_access_log
/var/log/containers/httpd/horizon/ssl_request_log
/var/log/containers/httpd/horizon/horizon_access.log
/var/log/containers/httpd/keystone/error_log
/var/log/containers/httpd/keystone/keystone_wsgi_main_error.log
/var/log/containers/httpd/keystone/keystone_wsgi_admin_error.log
/var/log/containers/httpd/keystone/keystone_wsgi_main_access.log
/var/log/containers/httpd/keystone/keystone_wsgi_admin_access.log
/var/log/containers/httpd/nova-api/error_log
/var/log/containers/httpd/nova-api/nova_api_wsgi_error.log
/var/log/containers/httpd/nova-api/nova_api_wsgi_access.log
/var/log/containers/httpd/nova-placement/error_log
/var/log/containers/httpd/nova-placement/placement_wsgi_error.log
/var/log/containers/httpd/nova-placement/placement_wsgi_access.log
/var/log/containers/httpd/panko-api/error_log
/var/log/containers/httpd/panko-api/panko_wsgi_error.log
/var/log/containers/httpd/panko-api/panko_wsgi_access.log
/var/log/containers/ceilometer/agent-notification.log
/var/log/containers/ceilometer/ceilometer-upgrade.log
/var/log/containers/glance/api.log
/var/log/containers/gnocchi/gnocchi-upgrade.log
/var/log/containers/gnocchi/gnocchi-metricd.log
/var/log/containers/gnocchi/gnocchi-statsd.log
/var/log/containers/gnocchi/app.log
/var/log/containers/heat/heat_api.log
/var/log/containers/heat/heat-engine.log
/var/log/containers/heat/heat_api_cfn.log
/var/log/containers/horizon/horizon.log
/var/log/containers/keystone/keystone.log
/var/log/containers/nova/nova-manage.log
/var/log/containers/nova/nova-scheduler.log
/var/log/containers/nova/nova-consoleauth.log
/var/log/containers/nova/nova-novncproxy.log
/var/log/containers/nova/nova-conductor.log
/var/log/containers/nova/nova-api-metadata.log
/var/log/containers/nova/nova-api.log
/var/log/containers/nova/nova-placement-api.log
/var/log/containers/panko/panko-dbsync.log
/var/log/containers/panko/app.log


For example looking at the above list. How does the user find the log of keystone_cron container?
There should be an easy way to correlate between a container and its log for all containers.

Comment 1 Dan Prince 2017-10-18 13:53:25 UTC
We could make use of the Ansible upgrade framework to handle this during the upgrade by doing the following for each service:

1) Move the existing logs for any baremetal services into a subdirectory. So /var/log/keystone/*.log gets moved to /var/log/keystone/backup/*.log

2) Add a README file in the /var/log/<service> directory that points to the new log file location in /var/log/containers/<service>. This will also mention that the old log files above have been moved into the 'backup' directory.

----

The above 2 things should help users who upgrade and new users know where to look for the new logs in /var/log/containers. By moving the old logs to backup directories we prevent confusion from SOS reports or other tools that might grab the old files.

We should also be more explicit about this new log location in our documentation.

Additionally, we should rename any ambiguous log files in /var/log/containers (app.log for example) to more closely match the service names.

A combination of the above should greatly help on this front.

Comment 4 Omri Hochman 2017-11-29 14:28:47 UTC
It's too late to get into osp12 GA --> should be in zstream

Comment 16 errata-xmlrpc 2018-08-20 12:58:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2331


Note You need to log in before you can comment on or make changes to this bug.