Bug 1640536

Summary:	gnocchi-statsd container heath-check failed and empty docker logs for gnocchi-statsd container
Product:	Red Hat OpenStack	Reporter:	Sasha Smolyak <ssmolyak>
Component:	openstack-tripleo-common	Assignee:	Martin Magr <mmagr>
Status:	CLOSED ERRATA	QA Contact:	Sasha Smolyak <ssmolyak>
Severity:	high	Docs Contact:	Andrew Burden <aburden>
Priority:	high
Version:	13.0 (Queens)	CC:	jschluet, lmarsh, m.andre, mburns, mmagr, slinaber
Target Milestone:	z3	Keywords:	Triaged, ZStream
Target Release:	13.0 (Queens)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-tripleo-common-8.6.6-2.el7ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-11-13 22:31:14 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Sasha Smolyak 2018-10-18 09:55:27 UTC

*This bug is a clone of https://bugzilla.redhat.com/show_bug.cgi?id=1608485*

Description of problem:
OSP13

483e101fd265        192.168.24.1:8787/rhosp14/openstack-gnocchi-statsd:2018-07-09.1              "kolla_start"            24 hours ago        Up 24 hours (unhealthy)                       gnocchi_statsd


[heat-admin@controller-0 ~]$ sudo docker exec -it gnocchi_statsd /bin/bash
()[gnocchi@controller-0 /]$ /openstack/healthcheck
There is no gnocchi-statsd process with opened Redis ports (6379) running in the container


()[gnocchi@controller-0 /]$ ps -aux |grep gnocchi       
gnocchi        1  0.0  0.2 523740 69728 ?        Ssl  Jul24   0:03 /usr/bin/python2 /usr/bin/gnocchi-statsd
gnocchi    29816  0.3  0.0  11816  1724 ?        Ss   15:29   0:00 /bin/bash
gnocchi    29846  0.0  0.0  51708  1704 ?        R+   15:29   0:00 ps -aux
gnocchi    29847  0.0  0.0   9088   660 ?        S+   15:29   0:00 grep gnocchi
()[gnocchi@controller-0 /]$ exit
exit
[heat-admin@controller-0 ~]$ ps faux |grep gnocchi-statsd
42416     126422  0.0  0.2 523740 69728 ?        Ssl  Jul24   0:03      |   \_ /usr/bin/python2 /usr/bin/gnocchi-statsd
heat-ad+  210390  0.0  0.0 112704   968 pts/0    S+   15:31   0:00              \_ grep --color=auto gnocchi-statsd
[heat-admin@controller-0 ~]$ ls /var/log/containers/gnocchi/



[heat-admin@controller-0 ~]$ sudo docker logs gnocchi_statsd 
[heat-admin@controller-0 ~]$ 

[heat-admin@controller-0 ~]$ cat /var/log/containers/gnocchi/gnocchi-statsd.log.1 
2018-07-24 14:37:03,604 [1] WARNING  oslo_config.cfg: Option "coordination_url" from group "storage" is deprecated. Use option "coordination_url" from group "DEFAULT".
2018-07-24 14:37:03,604 [1] INFO     gnocchi.service: Gnocchi version 4.2.6
2018-07-24 14:37:03,604 [1] DEBUG    gnocchi.service: ********************************************************************************
2018-07-24 14:37:03,604 [1] DEBUG    gnocchi.service: Configuration options gathered from:
2018-07-24 14:37:03,605 [1] DEBUG    gnocchi.service: command line args: []
2018-07-24 14:37:03,605 [1] DEBUG    gnocchi.service: config files: ['/usr/share/gnocchi/gnocchi-dist.conf', '/etc/gnocchi/gnocchi.conf']
2018-07-24 14:37:03,605 [1] DEBUG    gnocchi.service: ================================================================================
2018-07-24 14:37:03,605 [1] DEBUG    gnocchi.service: config_dir                     = []
2018-07-24 14:37:03,605 [1] DEBUG    gnocchi.service: config_file                    = ['/usr/share/gnocchi/gnocchi-dist.conf', '/etc/gnocchi/gnocchi.conf']
2018-07-24 14:37:03,605 [1] DEBUG    gnocchi.service: coordination_url               = ****
2018-07-24 14:37:03,605 [1] DEBUG    gnocchi.service: debug                          = True
2018-07-24 14:37:03,605 [1] DEBUG    gnocchi.service: log_dir                        = /var/log/gnocchi
2018-07-24 14:37:03,605 [1] DEBUG    gnocchi.service: log_file                       = None
2018-07-24 14:37:03,606 [1] DEBUG    gnocchi.service: parallel_operations            = 8
2018-07-24 14:37:03,606 [1] DEBUG    gnocchi.service: syslog_log_facility            = user
2018-07-24 14:37:03,606 [1] DEBUG    gnocchi.service: use_journal                    = False
2018-07-24 14:37:03,606 [1] DEBUG    gnocchi.service: use_syslog                     = False
2018-07-24 14:37:03,606 [1] DEBUG    gnocchi.service: verbose                        = True
2018-07-24 14:37:03,606 [1] DEBUG    gnocchi.service: statsd.archive_policy_name     = low
2018-07-24 14:37:03,607 [1] DEBUG    gnocchi.service: statsd.creator                 = None
2018-07-24 14:37:03,607 [1] DEBUG    gnocchi.service: statsd.flush_delay             = 10.0
2018-07-24 14:37:03,607 [1] DEBUG    gnocchi.service: statsd.host                    = 0.0.0.0
2018-07-24 14:37:03,607 [1] DEBUG    gnocchi.service: statsd.port                    = 8125
2018-07-24 14:37:03,608 [1] DEBUG    gnocchi.service: statsd.resource_id             = 0a8b55df-f90f-491c-8cb9-7cdecec6fc26
2018-07-24 14:37:03,608 [1] DEBUG    gnocchi.service: incoming.ceph_conffile         = /etc/ceph/ceph.conf
2018-07-24 14:37:03,608 [1] DEBUG    gnocchi.service: incoming.ceph_keyring          = /etc/ceph/ceph.client.openstack.keyring
2018-07-24 14:37:03,609 [1] DEBUG    gnocchi.service: incoming.ceph_pool             = metrics
2018-07-24 14:37:03,609 [1] DEBUG    gnocchi.service: incoming.ceph_secret           = ****
2018-07-24 14:37:03,609 [1] DEBUG    gnocchi.service: incoming.ceph_timeout          = 30
2018-07-24 14:37:03,610 [1] DEBUG    gnocchi.service: incoming.ceph_username         = openstack
2018-07-24 14:37:03,610 [1] DEBUG    gnocchi.service: incoming.driver                = redis
2018-07-24 14:37:03,610 [1] DEBUG    gnocchi.service: incoming.file_basepath         = /var/lib/gnocchi
2018-07-24 14:37:03,611 [1] DEBUG    gnocchi.service: incoming.redis_url             = redis://:J1VAPNLI89VoioU23cvMENQFB.1.14:6379/
2018-07-24 14:37:03,611 [1] DEBUG    gnocchi.service: incoming.s3_access_key_id      = 
2018-07-24 14:37:03,611 [1] DEBUG    gnocchi.service: incoming.s3_bucket_prefix      = gnocchi
2018-07-24 14:37:03,611 [1] DEBUG    gnocchi.service: incoming.s3_check_consistency_timeout = 60.0
2018-07-24 14:37:03,612 [1] DEBUG    gnocchi.service: incoming.s3_endpoint_url       = 
2018-07-24 14:37:03,612 [1] DEBUG    gnocchi.service: incoming.s3_max_pool_connections = 50
2018-07-24 14:37:03,612 [1] DEBUG    gnocchi.service: incoming.s3_region_name        = 
2018-07-24 14:37:03,612 [1] DEBUG    gnocchi.service: incoming.s3_secret_access_key  = 
2018-07-24 14:37:03,613 [1] DEBUG    gnocchi.service: incoming.swift_auth_insecure   = False
2018-07-24 14:37:03,613 [1] DEBUG    gnocchi.service: incoming.swift_auth_version    = 1
2018-07-24 14:37:03,613 [1] DEBUG    gnocchi.service: incoming.swift_authurl         = http://localhost:8080/auth/v1.0
2018-07-24 14:37:03,614 [1] DEBUG    gnocchi.service: incoming.swift_cacert          = 
2018-07-24 14:37:03,614 [1] DEBUG    gnocchi.service: incoming.swift_container_prefix = gnocchi
2018-07-24 14:37:03,614 [1] DEBUG    gnocchi.service: incoming.swift_endpoint_type   = publicURL
2018-07-24 14:37:03,615 [1] DEBUG    gnocchi.service: incoming.swift_key             = ****
2018-07-24 14:37:03,615 [1] DEBUG    gnocchi.service: incoming.swift_preauthtoken    = ****
2018-07-24 14:37:03,615 [1] DEBUG    gnocchi.service: incoming.swift_project_domain_name = Default
2018-07-24 14:37:03,615 [1] DEBUG    gnocchi.service: incoming.swift_project_name    = 
2018-07-24 14:37:03,616 [1] DEBUG    gnocchi.service: incoming.swift_region          = 
2018-07-24 14:37:03,616 [1] DEBUG    gnocchi.service: incoming.swift_service_type    = object-store
2018-07-24 14:37:03,616 [1] DEBUG    gnocchi.service: incoming.swift_timeout         = 300
2018-07-24 14:37:03,616 [1] DEBUG    gnocchi.service: incoming.swift_url             = 
2018-07-24 14:37:03,617 [1] DEBUG    gnocchi.service: incoming.swift_user            = admin:admin
2018-07-24 14:37:03,617 [1] DEBUG    gnocchi.service: incoming.swift_user_domain_name = Default
2018-07-24 14:37:03,617 [1] DEBUG    gnocchi.service: metricd.greedy                 = True
2018-07-24 14:37:03,618 [1] DEBUG    gnocchi.service: metricd.metric_cleanup_delay   = 300
2018-07-24 14:37:03,618 [1] DEBUG    gnocchi.service: metricd.metric_processing_delay = 30
2018-07-24 14:37:03,618 [1] DEBUG    gnocchi.service: metricd.metric_reporting_delay = 120
2018-07-24 14:37:03,618 [1] DEBUG    gnocchi.service: metricd.processing_replicas    = 3
2018-07-24 14:37:03,618 [1] DEBUG    gnocchi.service: metricd.workers                = 4
2018-07-24 14:37:03,618 [1] DEBUG    gnocchi.service: database.backend               = sqlalchemy
2018-07-24 14:37:03,619 [1] DEBUG    gnocchi.service: database.connection            = ****
2018-07-24 14:37:03,619 [1] DEBUG    gnocchi.service: database.connection_debug      = 0
2018-07-24 14:37:03,619 [1] DEBUG    gnocchi.service: database.connection_parameters = 
2018-07-24 14:37:03,619 [1] DEBUG    gnocchi.service: database.connection_recycle_time = 3600
2018-07-24 14:37:03,619 [1] DEBUG    gnocchi.service: database.connection_trace      = False
2018-07-24 14:37:03,619 [1] DEBUG    gnocchi.service: database.db_inc_retry_interval = True
2018-07-24 14:37:03,619 [1] DEBUG    gnocchi.service: database.db_max_retries        = 20
2018-07-24 14:37:03,620 [1] DEBUG    gnocchi.service: database.db_max_retry_interval = 10
2018-07-24 14:37:03,620 [1] DEBUG    gnocchi.service: database.db_retry_interval     = 1
2018-07-24 14:37:03,620 [1] DEBUG    gnocchi.service: database.max_overflow          = 50
2018-07-24 14:37:03,620 [1] DEBUG    gnocchi.service: database.max_pool_size         = 5
2018-07-24 14:37:03,620 [1] DEBUG    gnocchi.service: database.max_retries           = 10
2018-07-24 14:37:03,620 [1] DEBUG    gnocchi.service: database.min_pool_size         = 1
2018-07-24 14:37:03,620 [1] DEBUG    gnocchi.service: database.mysql_enable_ndb      = False
2018-07-24 14:37:03,621 [1] DEBUG    gnocchi.service: database.mysql_sql_mode        = TRADITIONAL
2018-07-24 14:37:03,621 [1] DEBUG    gnocchi.service: database.pool_timeout          = None
2018-07-24 14:37:03,621 [1] DEBUG    gnocchi.service: database.retry_interval        = 10
2018-07-24 14:37:03,621 [1] DEBUG    gnocchi.service: database.slave_connection      = ****
2018-07-24 14:37:03,621 [1] DEBUG    gnocchi.service: database.sqlite_synchronous    = True
2018-07-24 14:37:03,621 [1] DEBUG    gnocchi.service: database.use_db_reconnect      = False
2018-07-24 14:37:03,621 [1] DEBUG    gnocchi.service: storage.ceph_conffile          = /etc/ceph/ceph.conf
2018-07-24 14:37:03,622 [1] DEBUG    gnocchi.service: storage.ceph_keyring           = /etc/ceph/ceph.client.openstack.keyring
2018-07-24 14:37:03,622 [1] DEBUG    gnocchi.service: storage.ceph_pool              = metrics
2018-07-24 14:37:03,622 [1] DEBUG    gnocchi.service: storage.ceph_secret            = ****
2018-07-24 14:37:03,622 [1] DEBUG    gnocchi.service: storage.ceph_timeout           = 30
2018-07-24 14:37:03,622 [1] DEBUG    gnocchi.service: storage.ceph_username          = openstack
2018-07-24 14:37:03,622 [1] DEBUG    gnocchi.service: storage.driver                 = ceph
2018-07-24 14:37:03,622 [1] DEBUG    gnocchi.service: storage.file_basepath          = /var/lib/gnocchi
2018-07-24 14:37:03,622 [1] DEBUG    gnocchi.service: storage.redis_url              = redis://localhost:6379/
2018-07-24 14:37:03,622 [1] DEBUG    gnocchi.service: storage.s3_access_key_id       = None
2018-07-24 14:37:03,622 [1] DEBUG    gnocchi.service: storage.s3_bucket_prefix       = gnocchi
2018-07-24 14:37:03,623 [1] DEBUG    gnocchi.service: storage.s3_check_consistency_timeout = 60.0
2018-07-24 14:37:03,623 [1] DEBUG    gnocchi.service: storage.s3_endpoint_url        = None
2018-07-24 14:37:03,623 [1] DEBUG    gnocchi.service: storage.s3_max_pool_connections = 50
2018-07-24 14:37:03,623 [1] DEBUG    gnocchi.service: storage.s3_region_name         = None
2018-07-24 14:37:03,623 [1] DEBUG    gnocchi.service: storage.s3_secret_access_key   = None
2018-07-24 14:37:03,623 [1] DEBUG    gnocchi.service: storage.swift_auth_insecure    = False
2018-07-24 14:37:03,623 [1] DEBUG    gnocchi.service: storage.swift_auth_version     = 1
2018-07-24 14:37:03,623 [1] DEBUG    gnocchi.service: storage.swift_authurl          = http://localhost:8080/auth/v1.0
2018-07-24 14:37:03,623 [1] DEBUG    gnocchi.service: storage.swift_cacert           = None
2018-07-24 14:37:03,623 [1] DEBUG    gnocchi.service: storage.swift_container_prefix = gnocchi
2018-07-24 14:37:03,624 [1] DEBUG    gnocchi.service: storage.swift_endpoint_type    = publicURL
2018-07-24 14:37:03,624 [1] DEBUG    gnocchi.service: storage.swift_key              = ****
2018-07-24 14:37:03,624 [1] DEBUG    gnocchi.service: storage.swift_preauthtoken     = ****
2018-07-24 14:37:03,624 [1] DEBUG    gnocchi.service: storage.swift_project_domain_name = Default
2018-07-24 14:37:03,624 [1] DEBUG    gnocchi.service: storage.swift_project_name     = None
2018-07-24 14:37:03,624 [1] DEBUG    gnocchi.service: storage.swift_region           = None
2018-07-24 14:37:03,625 [1] DEBUG    gnocchi.service: storage.swift_service_type     = object-store
2018-07-24 14:37:03,625 [1] DEBUG    gnocchi.service: storage.swift_timeout          = 300
2018-07-24 14:37:03,625 [1] DEBUG    gnocchi.service: storage.swift_url              = None
2018-07-24 14:37:03,625 [1] DEBUG    gnocchi.service: storage.swift_user             = admin:admin
2018-07-24 14:37:03,625 [1] DEBUG    gnocchi.service: storage.swift_user_domain_name = Default
2018-07-24 14:37:03,625 [1] DEBUG    gnocchi.service: indexer.url                    = ****
2018-07-24 14:37:03,625 [1] DEBUG    gnocchi.service: api.auth_mode                  = keystone
2018-07-24 14:37:03,626 [1] DEBUG    gnocchi.service: api.host                       = 0.0.0.0
2018-07-24 14:37:03,626 [1] DEBUG    gnocchi.service: api.max_limit                  = 1000
2018-07-24 14:37:03,626 [1] DEBUG    gnocchi.service: api.operation_timeout          = 10
2018-07-24 14:37:03,626 [1] DEBUG    gnocchi.service: api.paste_config               = api-paste.ini
2018-07-24 14:37:03,626 [1] DEBUG    gnocchi.service: api.port                       = 8041
2018-07-24 14:37:03,626 [1] DEBUG    gnocchi.service: api.uwsgi_mode                 = http
2018-07-24 14:37:03,626 [1] DEBUG    gnocchi.service: archive_policy.default_aggregation_methods = ['mean', 'min', 'max', 'sum', 'std', 'count']
2018-07-24 14:37:03,626 [1] DEBUG    gnocchi.service: ********************************************************************************
2018-07-24 14:37:04,361 [1] DEBUG    gnocchi.statsd: Resource 0a8b55df-f90f-491c-8cb9-7cdecec6fc26 already exists
2018-07-24 14:37:04,388 [1] INFO     gnocchi.statsd: Started on 0.0.0.0:8125
2018-07-24 14:37:04,389 [1] INFO     gnocchi.statsd: Flush delay: 10 seconds


Version-Release number of selected component (if applicable):
OSP14 puddle- 2018-07-09.1
openstack-gnocchi-statsd:2018-07-09.1 container
How reproducible:


Steps to Reproduce:
1.Deploy OSP14 containerized undercloud & overcloid
2. Check docker ps output


Actual results:
gnocchi-statsd container heath_check failed

Expected results:
all containers healthy

Additional info:

Comment 6 Sasha Smolyak 2018-11-01 12:12:12 UTC

puddle 2018-10-30.1
[heat-admin@controller-0 ~]$ sudo docker ps | grep gnocchi
df150f683033        192.168.24.1:8787/rhosp13/openstack-gnocchi-metricd:2018-10-30.1             "kolla_start"            18 hours ago        Up 18 hours (healthy)                         gnocchi_metricd
88dd4a29e864        192.168.24.1:8787/rhosp13/openstack-gnocchi-statsd:2018-10-30.1              "kolla_start"            18 hours ago        Up 18 hours (unhealthy)                       gnocchi_statsd
abe8b09f9f8f        192.168.24.1:8787/rhosp13/openstack-gnocchi-api:2018-10-30.1                 "kolla_start"            18 hours ago        Up 18 hours (healthy)                         gnocchi_api


[heat-admin@controller-0 ~]$ cat /var/log/containers/gnocchi/gnocchi-statsd.log.1
2018-10-31 17:18:37,051 [1] WARNING  oslo_config.cfg: Option "coordination_url" from group "storage" is deprecated. Use option "coordination_url" from group "DEFAULT".
2018-10-31 17:18:37,051 [1] INFO     gnocchi.service: Gnocchi version 4.2.5
2018-10-31 17:18:37,472 [1] INFO     gnocchi.statsd: Started on 0.0.0.0:8125
2018-10-31 17:18:37,472 [1] INFO     gnocchi.statsd: Flush delay: 10 seconds

Comment 7 Jon Schlueter 2018-11-06 12:23:50 UTC

false failure as this was GA content not the batch update please retest

Comment 11 Sasha Smolyak 2018-11-12 07:33:57 UTC

Container is healthy, no errors in log. Fixed

Comment 13 errata-xmlrpc 2018-11-13 22:31:14 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3587