Red Hat Bugzilla – Bug 1502860
rhosp-director: difficult to map certain containers to their logs.
Last modified: 2018-08-20 08:59:12 EDT
rhosp-director: difficult to map certain containers to their logs. Environment: openstack-tripleo-heat-templates-7.0.2-0.20171007062244.el7ost.noarch instack-undercloud-7.4.2-0.20171006021856.el7ost.noarch openstack-puppet-modules-11.0.0-0.20170828113154.el7ost.noarch These are the names of containers on a controller: gnocchi_api gnocchi_statsd gnocchi_metricd panko_api nova_metadata nova_api glance_api swift_account_server aodh_listener swift_container_auditor heat_api_cron swift_object_expirer swift_object_updater swift_container_replicator swift_account_auditor logrotate_crond heat_api_cfn nova_conductor swift_object_replicator swift_container_server heat_engine aodh_api swift_rsync nova_vnc_proxy ceilometer_agent_notification swift_account_reaper nova_consoleauth nova_api_cron aodh_notifier ceilometer_agent_central swift_account_replicator swift_object_auditor heat_api swift_proxy swift_object_server nova_scheduler swift_container_updater aodh_evaluator keystone_cron keystone nova_placement horizon haproxy-bundle-docker-0 redis-bundle-docker-0 galera-bundle-docker-0 rabbitmq-bundle-docker-0 clustercheck ceph-mon-overcloud-controller-0 memcached These are the container logs: /var/log/containers/aodh/aodh-dbsync.log /var/log/containers/aodh/aodh-evaluator.log /var/log/containers/aodh/aodh-notifier.log /var/log/containers/aodh/app.log /var/log/containers/aodh/aodh-listener.log /var/log/containers/httpd/aodh-api/error_log /var/log/containers/httpd/aodh-api/aodh_wsgi_error.log /var/log/containers/httpd/aodh-api/aodh_wsgi_access.log /var/log/containers/httpd/gnocchi-api/error_log /var/log/containers/httpd/gnocchi-api/gnocchi_wsgi_error.log /var/log/containers/httpd/gnocchi-api/gnocchi_wsgi_access.log /var/log/containers/httpd/heat-api/error_log /var/log/containers/httpd/heat-api/heat_api_wsgi_error.log /var/log/containers/httpd/heat-api/heat_api_wsgi_access.log /var/log/containers/httpd/heat-api-cfn/error_log /var/log/containers/httpd/heat-api-cfn/heat_api_cfn_wsgi_error.log /var/log/containers/httpd/heat-api-cfn/heat_api_cfn_wsgi_access.log /var/log/containers/httpd/horizon/error_log /var/log/containers/httpd/horizon/ssl_error_log /var/log/containers/httpd/horizon/horizon_error.log /var/log/containers/httpd/horizon/ssl_access_log /var/log/containers/httpd/horizon/ssl_request_log /var/log/containers/httpd/horizon/horizon_access.log /var/log/containers/httpd/keystone/error_log /var/log/containers/httpd/keystone/keystone_wsgi_main_error.log /var/log/containers/httpd/keystone/keystone_wsgi_admin_error.log /var/log/containers/httpd/keystone/keystone_wsgi_main_access.log /var/log/containers/httpd/keystone/keystone_wsgi_admin_access.log /var/log/containers/httpd/nova-api/error_log /var/log/containers/httpd/nova-api/nova_api_wsgi_error.log /var/log/containers/httpd/nova-api/nova_api_wsgi_access.log /var/log/containers/httpd/nova-placement/error_log /var/log/containers/httpd/nova-placement/placement_wsgi_error.log /var/log/containers/httpd/nova-placement/placement_wsgi_access.log /var/log/containers/httpd/panko-api/error_log /var/log/containers/httpd/panko-api/panko_wsgi_error.log /var/log/containers/httpd/panko-api/panko_wsgi_access.log /var/log/containers/ceilometer/agent-notification.log /var/log/containers/ceilometer/ceilometer-upgrade.log /var/log/containers/glance/api.log /var/log/containers/gnocchi/gnocchi-upgrade.log /var/log/containers/gnocchi/gnocchi-metricd.log /var/log/containers/gnocchi/gnocchi-statsd.log /var/log/containers/gnocchi/app.log /var/log/containers/heat/heat_api.log /var/log/containers/heat/heat-engine.log /var/log/containers/heat/heat_api_cfn.log /var/log/containers/horizon/horizon.log /var/log/containers/keystone/keystone.log /var/log/containers/nova/nova-manage.log /var/log/containers/nova/nova-scheduler.log /var/log/containers/nova/nova-consoleauth.log /var/log/containers/nova/nova-novncproxy.log /var/log/containers/nova/nova-conductor.log /var/log/containers/nova/nova-api-metadata.log /var/log/containers/nova/nova-api.log /var/log/containers/nova/nova-placement-api.log /var/log/containers/panko/panko-dbsync.log /var/log/containers/panko/app.log For example looking at the above list. How does the user find the log of keystone_cron container? There should be an easy way to correlate between a container and its log for all containers.
We could make use of the Ansible upgrade framework to handle this during the upgrade by doing the following for each service: 1) Move the existing logs for any baremetal services into a subdirectory. So /var/log/keystone/*.log gets moved to /var/log/keystone/backup/*.log 2) Add a README file in the /var/log/<service> directory that points to the new log file location in /var/log/containers/<service>. This will also mention that the old log files above have been moved into the 'backup' directory. ---- The above 2 things should help users who upgrade and new users know where to look for the new logs in /var/log/containers. By moving the old logs to backup directories we prevent confusion from SOS reports or other tools that might grab the old files. We should also be more explicit about this new log location in our documentation. Additionally, we should rename any ambiguous log files in /var/log/containers (app.log for example) to more closely match the service names. A combination of the above should greatly help on this front.
It's too late to get into osp12 GA --> should be in zstream
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2331