Description of problem: OSP14 483e101fd265 192.168.24.1:8787/rhosp14/openstack-gnocchi-statsd:2018-07-09.1 "kolla_start" 24 hours ago Up 24 hours (unhealthy) gnocchi_statsd [heat-admin@controller-0 ~]$ sudo docker exec -it gnocchi_statsd /bin/bash ()[gnocchi@controller-0 /]$ /openstack/healthcheck There is no gnocchi-statsd process with opened Redis ports (6379) running in the container ()[gnocchi@controller-0 /]$ ps -aux |grep gnocchi gnocchi 1 0.0 0.2 523740 69728 ? Ssl Jul24 0:03 /usr/bin/python2 /usr/bin/gnocchi-statsd gnocchi 29816 0.3 0.0 11816 1724 ? Ss 15:29 0:00 /bin/bash gnocchi 29846 0.0 0.0 51708 1704 ? R+ 15:29 0:00 ps -aux gnocchi 29847 0.0 0.0 9088 660 ? S+ 15:29 0:00 grep gnocchi ()[gnocchi@controller-0 /]$ exit exit [heat-admin@controller-0 ~]$ ps faux |grep gnocchi-statsd 42416 126422 0.0 0.2 523740 69728 ? Ssl Jul24 0:03 | \_ /usr/bin/python2 /usr/bin/gnocchi-statsd heat-ad+ 210390 0.0 0.0 112704 968 pts/0 S+ 15:31 0:00 \_ grep --color=auto gnocchi-statsd [heat-admin@controller-0 ~]$ ls /var/log/containers/gnocchi/ [heat-admin@controller-0 ~]$ sudo docker logs gnocchi_statsd [heat-admin@controller-0 ~]$ [heat-admin@controller-0 ~]$ cat /var/log/containers/gnocchi/gnocchi-statsd.log.1 2018-07-24 14:37:03,604 [1] WARNING oslo_config.cfg: Option "coordination_url" from group "storage" is deprecated. Use option "coordination_url" from group "DEFAULT". 2018-07-24 14:37:03,604 [1] INFO gnocchi.service: Gnocchi version 4.2.6 2018-07-24 14:37:03,604 [1] DEBUG gnocchi.service: ******************************************************************************** 2018-07-24 14:37:03,604 [1] DEBUG gnocchi.service: Configuration options gathered from: 2018-07-24 14:37:03,605 [1] DEBUG gnocchi.service: command line args: [] 2018-07-24 14:37:03,605 [1] DEBUG gnocchi.service: config files: ['/usr/share/gnocchi/gnocchi-dist.conf', '/etc/gnocchi/gnocchi.conf'] 2018-07-24 14:37:03,605 [1] DEBUG gnocchi.service: ================================================================================ 2018-07-24 14:37:03,605 [1] DEBUG gnocchi.service: config_dir = [] 2018-07-24 14:37:03,605 [1] DEBUG gnocchi.service: config_file = ['/usr/share/gnocchi/gnocchi-dist.conf', '/etc/gnocchi/gnocchi.conf'] 2018-07-24 14:37:03,605 [1] DEBUG gnocchi.service: coordination_url = **** 2018-07-24 14:37:03,605 [1] DEBUG gnocchi.service: debug = True 2018-07-24 14:37:03,605 [1] DEBUG gnocchi.service: log_dir = /var/log/gnocchi 2018-07-24 14:37:03,605 [1] DEBUG gnocchi.service: log_file = None 2018-07-24 14:37:03,606 [1] DEBUG gnocchi.service: parallel_operations = 8 2018-07-24 14:37:03,606 [1] DEBUG gnocchi.service: syslog_log_facility = user 2018-07-24 14:37:03,606 [1] DEBUG gnocchi.service: use_journal = False 2018-07-24 14:37:03,606 [1] DEBUG gnocchi.service: use_syslog = False 2018-07-24 14:37:03,606 [1] DEBUG gnocchi.service: verbose = True 2018-07-24 14:37:03,606 [1] DEBUG gnocchi.service: statsd.archive_policy_name = low 2018-07-24 14:37:03,607 [1] DEBUG gnocchi.service: statsd.creator = None 2018-07-24 14:37:03,607 [1] DEBUG gnocchi.service: statsd.flush_delay = 10.0 2018-07-24 14:37:03,607 [1] DEBUG gnocchi.service: statsd.host = 0.0.0.0 2018-07-24 14:37:03,607 [1] DEBUG gnocchi.service: statsd.port = 8125 2018-07-24 14:37:03,608 [1] DEBUG gnocchi.service: statsd.resource_id = 0a8b55df-f90f-491c-8cb9-7cdecec6fc26 2018-07-24 14:37:03,608 [1] DEBUG gnocchi.service: incoming.ceph_conffile = /etc/ceph/ceph.conf 2018-07-24 14:37:03,608 [1] DEBUG gnocchi.service: incoming.ceph_keyring = /etc/ceph/ceph.client.openstack.keyring 2018-07-24 14:37:03,609 [1] DEBUG gnocchi.service: incoming.ceph_pool = metrics 2018-07-24 14:37:03,609 [1] DEBUG gnocchi.service: incoming.ceph_secret = **** 2018-07-24 14:37:03,609 [1] DEBUG gnocchi.service: incoming.ceph_timeout = 30 2018-07-24 14:37:03,610 [1] DEBUG gnocchi.service: incoming.ceph_username = openstack 2018-07-24 14:37:03,610 [1] DEBUG gnocchi.service: incoming.driver = redis 2018-07-24 14:37:03,610 [1] DEBUG gnocchi.service: incoming.file_basepath = /var/lib/gnocchi 2018-07-24 14:37:03,611 [1] DEBUG gnocchi.service: incoming.redis_url = redis://:J1VAPNLI89VoioU23cvMENQFB.1.14:6379/ 2018-07-24 14:37:03,611 [1] DEBUG gnocchi.service: incoming.s3_access_key_id = 2018-07-24 14:37:03,611 [1] DEBUG gnocchi.service: incoming.s3_bucket_prefix = gnocchi 2018-07-24 14:37:03,611 [1] DEBUG gnocchi.service: incoming.s3_check_consistency_timeout = 60.0 2018-07-24 14:37:03,612 [1] DEBUG gnocchi.service: incoming.s3_endpoint_url = 2018-07-24 14:37:03,612 [1] DEBUG gnocchi.service: incoming.s3_max_pool_connections = 50 2018-07-24 14:37:03,612 [1] DEBUG gnocchi.service: incoming.s3_region_name = 2018-07-24 14:37:03,612 [1] DEBUG gnocchi.service: incoming.s3_secret_access_key = 2018-07-24 14:37:03,613 [1] DEBUG gnocchi.service: incoming.swift_auth_insecure = False 2018-07-24 14:37:03,613 [1] DEBUG gnocchi.service: incoming.swift_auth_version = 1 2018-07-24 14:37:03,613 [1] DEBUG gnocchi.service: incoming.swift_authurl = http://localhost:8080/auth/v1.0 2018-07-24 14:37:03,614 [1] DEBUG gnocchi.service: incoming.swift_cacert = 2018-07-24 14:37:03,614 [1] DEBUG gnocchi.service: incoming.swift_container_prefix = gnocchi 2018-07-24 14:37:03,614 [1] DEBUG gnocchi.service: incoming.swift_endpoint_type = publicURL 2018-07-24 14:37:03,615 [1] DEBUG gnocchi.service: incoming.swift_key = **** 2018-07-24 14:37:03,615 [1] DEBUG gnocchi.service: incoming.swift_preauthtoken = **** 2018-07-24 14:37:03,615 [1] DEBUG gnocchi.service: incoming.swift_project_domain_name = Default 2018-07-24 14:37:03,615 [1] DEBUG gnocchi.service: incoming.swift_project_name = 2018-07-24 14:37:03,616 [1] DEBUG gnocchi.service: incoming.swift_region = 2018-07-24 14:37:03,616 [1] DEBUG gnocchi.service: incoming.swift_service_type = object-store 2018-07-24 14:37:03,616 [1] DEBUG gnocchi.service: incoming.swift_timeout = 300 2018-07-24 14:37:03,616 [1] DEBUG gnocchi.service: incoming.swift_url = 2018-07-24 14:37:03,617 [1] DEBUG gnocchi.service: incoming.swift_user = admin:admin 2018-07-24 14:37:03,617 [1] DEBUG gnocchi.service: incoming.swift_user_domain_name = Default 2018-07-24 14:37:03,617 [1] DEBUG gnocchi.service: metricd.greedy = True 2018-07-24 14:37:03,618 [1] DEBUG gnocchi.service: metricd.metric_cleanup_delay = 300 2018-07-24 14:37:03,618 [1] DEBUG gnocchi.service: metricd.metric_processing_delay = 30 2018-07-24 14:37:03,618 [1] DEBUG gnocchi.service: metricd.metric_reporting_delay = 120 2018-07-24 14:37:03,618 [1] DEBUG gnocchi.service: metricd.processing_replicas = 3 2018-07-24 14:37:03,618 [1] DEBUG gnocchi.service: metricd.workers = 4 2018-07-24 14:37:03,618 [1] DEBUG gnocchi.service: database.backend = sqlalchemy 2018-07-24 14:37:03,619 [1] DEBUG gnocchi.service: database.connection = **** 2018-07-24 14:37:03,619 [1] DEBUG gnocchi.service: database.connection_debug = 0 2018-07-24 14:37:03,619 [1] DEBUG gnocchi.service: database.connection_parameters = 2018-07-24 14:37:03,619 [1] DEBUG gnocchi.service: database.connection_recycle_time = 3600 2018-07-24 14:37:03,619 [1] DEBUG gnocchi.service: database.connection_trace = False 2018-07-24 14:37:03,619 [1] DEBUG gnocchi.service: database.db_inc_retry_interval = True 2018-07-24 14:37:03,619 [1] DEBUG gnocchi.service: database.db_max_retries = 20 2018-07-24 14:37:03,620 [1] DEBUG gnocchi.service: database.db_max_retry_interval = 10 2018-07-24 14:37:03,620 [1] DEBUG gnocchi.service: database.db_retry_interval = 1 2018-07-24 14:37:03,620 [1] DEBUG gnocchi.service: database.max_overflow = 50 2018-07-24 14:37:03,620 [1] DEBUG gnocchi.service: database.max_pool_size = 5 2018-07-24 14:37:03,620 [1] DEBUG gnocchi.service: database.max_retries = 10 2018-07-24 14:37:03,620 [1] DEBUG gnocchi.service: database.min_pool_size = 1 2018-07-24 14:37:03,620 [1] DEBUG gnocchi.service: database.mysql_enable_ndb = False 2018-07-24 14:37:03,621 [1] DEBUG gnocchi.service: database.mysql_sql_mode = TRADITIONAL 2018-07-24 14:37:03,621 [1] DEBUG gnocchi.service: database.pool_timeout = None 2018-07-24 14:37:03,621 [1] DEBUG gnocchi.service: database.retry_interval = 10 2018-07-24 14:37:03,621 [1] DEBUG gnocchi.service: database.slave_connection = **** 2018-07-24 14:37:03,621 [1] DEBUG gnocchi.service: database.sqlite_synchronous = True 2018-07-24 14:37:03,621 [1] DEBUG gnocchi.service: database.use_db_reconnect = False 2018-07-24 14:37:03,621 [1] DEBUG gnocchi.service: storage.ceph_conffile = /etc/ceph/ceph.conf 2018-07-24 14:37:03,622 [1] DEBUG gnocchi.service: storage.ceph_keyring = /etc/ceph/ceph.client.openstack.keyring 2018-07-24 14:37:03,622 [1] DEBUG gnocchi.service: storage.ceph_pool = metrics 2018-07-24 14:37:03,622 [1] DEBUG gnocchi.service: storage.ceph_secret = **** 2018-07-24 14:37:03,622 [1] DEBUG gnocchi.service: storage.ceph_timeout = 30 2018-07-24 14:37:03,622 [1] DEBUG gnocchi.service: storage.ceph_username = openstack 2018-07-24 14:37:03,622 [1] DEBUG gnocchi.service: storage.driver = ceph 2018-07-24 14:37:03,622 [1] DEBUG gnocchi.service: storage.file_basepath = /var/lib/gnocchi 2018-07-24 14:37:03,622 [1] DEBUG gnocchi.service: storage.redis_url = redis://localhost:6379/ 2018-07-24 14:37:03,622 [1] DEBUG gnocchi.service: storage.s3_access_key_id = None 2018-07-24 14:37:03,622 [1] DEBUG gnocchi.service: storage.s3_bucket_prefix = gnocchi 2018-07-24 14:37:03,623 [1] DEBUG gnocchi.service: storage.s3_check_consistency_timeout = 60.0 2018-07-24 14:37:03,623 [1] DEBUG gnocchi.service: storage.s3_endpoint_url = None 2018-07-24 14:37:03,623 [1] DEBUG gnocchi.service: storage.s3_max_pool_connections = 50 2018-07-24 14:37:03,623 [1] DEBUG gnocchi.service: storage.s3_region_name = None 2018-07-24 14:37:03,623 [1] DEBUG gnocchi.service: storage.s3_secret_access_key = None 2018-07-24 14:37:03,623 [1] DEBUG gnocchi.service: storage.swift_auth_insecure = False 2018-07-24 14:37:03,623 [1] DEBUG gnocchi.service: storage.swift_auth_version = 1 2018-07-24 14:37:03,623 [1] DEBUG gnocchi.service: storage.swift_authurl = http://localhost:8080/auth/v1.0 2018-07-24 14:37:03,623 [1] DEBUG gnocchi.service: storage.swift_cacert = None 2018-07-24 14:37:03,623 [1] DEBUG gnocchi.service: storage.swift_container_prefix = gnocchi 2018-07-24 14:37:03,624 [1] DEBUG gnocchi.service: storage.swift_endpoint_type = publicURL 2018-07-24 14:37:03,624 [1] DEBUG gnocchi.service: storage.swift_key = **** 2018-07-24 14:37:03,624 [1] DEBUG gnocchi.service: storage.swift_preauthtoken = **** 2018-07-24 14:37:03,624 [1] DEBUG gnocchi.service: storage.swift_project_domain_name = Default 2018-07-24 14:37:03,624 [1] DEBUG gnocchi.service: storage.swift_project_name = None 2018-07-24 14:37:03,624 [1] DEBUG gnocchi.service: storage.swift_region = None 2018-07-24 14:37:03,625 [1] DEBUG gnocchi.service: storage.swift_service_type = object-store 2018-07-24 14:37:03,625 [1] DEBUG gnocchi.service: storage.swift_timeout = 300 2018-07-24 14:37:03,625 [1] DEBUG gnocchi.service: storage.swift_url = None 2018-07-24 14:37:03,625 [1] DEBUG gnocchi.service: storage.swift_user = admin:admin 2018-07-24 14:37:03,625 [1] DEBUG gnocchi.service: storage.swift_user_domain_name = Default 2018-07-24 14:37:03,625 [1] DEBUG gnocchi.service: indexer.url = **** 2018-07-24 14:37:03,625 [1] DEBUG gnocchi.service: api.auth_mode = keystone 2018-07-24 14:37:03,626 [1] DEBUG gnocchi.service: api.host = 0.0.0.0 2018-07-24 14:37:03,626 [1] DEBUG gnocchi.service: api.max_limit = 1000 2018-07-24 14:37:03,626 [1] DEBUG gnocchi.service: api.operation_timeout = 10 2018-07-24 14:37:03,626 [1] DEBUG gnocchi.service: api.paste_config = api-paste.ini 2018-07-24 14:37:03,626 [1] DEBUG gnocchi.service: api.port = 8041 2018-07-24 14:37:03,626 [1] DEBUG gnocchi.service: api.uwsgi_mode = http 2018-07-24 14:37:03,626 [1] DEBUG gnocchi.service: archive_policy.default_aggregation_methods = ['mean', 'min', 'max', 'sum', 'std', 'count'] 2018-07-24 14:37:03,626 [1] DEBUG gnocchi.service: ******************************************************************************** 2018-07-24 14:37:04,361 [1] DEBUG gnocchi.statsd: Resource 0a8b55df-f90f-491c-8cb9-7cdecec6fc26 already exists 2018-07-24 14:37:04,388 [1] INFO gnocchi.statsd: Started on 0.0.0.0:8125 2018-07-24 14:37:04,389 [1] INFO gnocchi.statsd: Flush delay: 10 seconds Version-Release number of selected component (if applicable): OSP14 puddle- 2018-07-09.1 openstack-gnocchi-statsd:2018-07-09.1 container How reproducible: Steps to Reproduce: 1.Deploy OSP14 containerized undercloud & overcloid 2. Check docker ps output Actual results: gnocchi-statsd container heath_check failed Expected results: all containers healthy Additional info:
I am seeing this issue in an OSP13 environment I just deployed: [heat-admin@overcloud-controller-0 ~]$ sudo docker exec -it gnocchi_statsd /bin/bash ()[gnocchi@overcloud-controller-0 /]$ /openstack/healthcheck There is no gnocchi-statsd process with opened Redis ports (6379) running in the container ()[gnocchi@overcloud-controller-0 /]$ exit exit [heat-admin@overcloud-controller-0 ~]$ sudo docker inspect gnocchi_statsd --format='{{json .Config.Labels}}' | jq . { "version": "13.0", "vendor": "Red Hat, Inc.", "vcs-type": "git", "vcs-ref": "0ff1226f00696efc516d162e92402e37e29cd80c", "usage": "This image is very generic and does not serve a single use case. Use it as a base to build your own images.", "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp13/openstack-gnocchi-statsd/images/13.0-54", "summary": "Red Hat OpenStack Platform 13.0 gnocchi-statsd", "release": "54", "name": "rhosp13/openstack-gnocchi-statsd", "managed_by": "paunch", "config_id": "tripleo_step5", "config_data": "{\"start_order\": 1, \"healthcheck\": {\"test\": \"/openstack/healthcheck\"}, \"image\": \"172.16.0.11:8787/rhosp13/openstack-gnocchi-statsd:13.0-54\", \"environment\": [\"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\", \"TRIPLEO_CONFIG_HASH=25df471535b4d1a71ed45fafbc3ea770\"], \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/var/lib/kolla/config_files/gnocchi_statsd.json:/var/lib/kolla/config_files/config.json:ro\", \"/var/lib/config-data/puppet-generated/gnocchi/:/var/lib/kolla/config_files/src:ro\", \"/var/log/containers/gnocchi:/var/log/gnocchi\", \"/etc/ceph:/var/lib/kolla/config_files/src-ceph:ro\", \"/var/lib/gnocchi:/var/lib/gnocchi\"], \"net\": \"host\", \"privileged\": false, \"restart\": \"always\"}", "com.redhat.component": "openstack-gnocchi-statsd-container", "com.redhat.build-host": "osbs-cpt-011.ocp.osbs.upshift.eng.rdu2.redhat.com", "build-date": "2018-08-16T02:02:14.649726", "batch": "20180814.1", "authoritative-source-url": "registry.access.redhat.com", "architecture": "x86_64", "container_name": "gnocchi_statsd", "description": "Red Hat OpenStack Platform 13.0 gnocchi-statsd", "distribution-scope": "public", "io.k8s.description": "Red Hat OpenStack Platform 13.0 gnocchi-statsd", "io.k8s.display-name": "Red Hat OpenStack Platform 13.0 gnocchi-statsd", "io.openshift.expose-services": "", "io.openshift.tags": "rhosp osp openstack osp-13.0", "maintainer": "Red Hat, Inc." } [heat-admin@overcloud-controller-0 ~]$
From support case, one of the customer hit the exact issue on OSP 13. Could you backport it?
Do we want to clone it for RHOS13? In RHOS14 I can verify it it's fixed
Hi, This issue should be fixed in RHOSP13 also. We have several users to hit this issue and avoid the future inquires related to this issue.
Added a clone bug to RHOS13: https://bugzilla.redhat.com/show_bug.cgi?id=1640536 Verifying this one, it's fixed
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:0048