Description of problem: After upstream (pike) deployment with containers using quickstart the gnocchi-statsd is restarting Version-Release number of selected component (if applicable): How reproducible: always Steps to Reproduce: (1) from laptop or remote machine : fix the ssh-key by run: ssh-copy-id -i ~/.ssh/id_rsa.pub $VIRTHOST, where $VIRTHOST - your baremetal lab (2) to test run : ssh root@$VIRTHOST uname -a (3) wget https://raw.githubusercontent.com/openstack/tripleo-quickstart/master/quickstart.sh (4) bash quickstart.sh --install-deps (5) change target machine to latest (more fit CentOS7.3) repos --> http://pastebin.test.redhat.com/465894 (6) bash quickstart.sh --config /var/tmp/foo/config/general_config/containers_minimal.yml --working-dir /var/tmp/foo --teardown all --tags all --release master-tripleo-ci $VIRTHOST 7) [heat-admin@overcloud-controller-0 ~]$ sudo docker ps Actual results: 73e255c91c59 192.168.24.1:8787/tripleoupstream/centos-binary-gnocchi-statsd:latest "kolla_start" 18 hours ago Restarting (1) 4 hours ago gnocchi_statsd [heat-admin@overcloud-controller-0 ~]$ sudo docker exec -it gnocchi_statsd bash Error response from daemon: Container 73e255c91c59c27db6699ef593e8845490003ce711d4d903702e1a9d0fca685a is restarting, wait until the container is running Expected results: Container Status - Up Additional info:
This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.
might be related with : https://review.openstack.org/#/c/448077/
(undercloud) [stack@undercloud ~]$ rpm -qa |grep docker docker-common-1.12.6-11.el7.centos.x86_64 docker-1.12.6-11.el7.centos.x86_64 python-docker-py-1.10.6-1.el7.noarch docker-client-1.12.6-11.el7.centos.x86_64 python-heat-agent-docker-cmd-1.0.1-0.20170306124536.3961e3b.el7.centos.noarch docker-distribution-2.6.0-1.el7.x86_64 python-docker-pycreds-1.10.6-1.el7.noarch instack-6.0.1-0.20170310015230.db7123e.el7.centos.noarc instack-undercloud-6.0.0-0.20170317130016.58a4f6c.el7.centos.noarch heat-admin@overcloud-controller-0 ~]$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 508b01f42d5e 192.168.24.1:8787/tripleoupstream/centos-binary-nova-api:latest "kolla_start" 20 hours ago Up 20 hours nova_api 84ddcd8f0f6c 192.168.24.1:8787/tripleoupstream/centos-binary-heat-api-cfn:latest "kolla_start" 20 hours ago Up 20 hours heat_api_cfn a6784023413c 192.168.24.1:8787/tripleoupstream/centos-binary-neutron-server:latest "kolla_start" 20 hours ago Up 20 hours neutron_api e46e269d13d7 192.168.24.1:8787/tripleoupstream/centos-binary-aodh-listener:latest "kolla_start" 20 hours ago Up 20 hours aodh_listener 652f468a9f4f 192.168.24.1:8787/tripleoupstream/centos-binary-swift-container:latest "kolla_start" 20 hours ago Up 20 hours swift_container_auditor 41e5ee6ef6f5 192.168.24.1:8787/tripleoupstream/centos-binary-swift-proxy-server:latest "kolla_start" 20 hours ago Up 20 hours swift_object_expirer 3171133a3aa0 192.168.24.1:8787/tripleoupstream/centos-binary-swift-object:latest "kolla_start" 20 hours ago Up 20 hours swift_object_updater a2c107ce6b3e 192.168.24.1:8787/tripleoupstream/centos-binary-swift-container:latest "kolla_start" 20 hours ago Up 20 hours swift_container_replicator 2e942779bf39 192.168.24.1:8787/tripleoupstream/centos-binary-swift-account:latest "kolla_start" 20 hours ago Up 20 hours swift_account_auditor 134fa1d35bad 192.168.24.1:8787/tripleoupstream/centos-binary-aodh-api:latest "kolla_start" 20 hours ago Up 20 hours aodh-api 859bd6ce0520 192.168.24.1:8787/tripleoupstream/centos-binary-swift-account:latest "kolla_start" 20 hours ago Up 20 hours swift_account_server a5e028baf447 192.168.24.1:8787/tripleoupstream/centos-binary-nova-scheduler:latest "kolla_start" 20 hours ago Up 20 hours nova_scheduler 91486afbb001 192.168.24.1:8787/tripleoupstream/centos-binary-gnocchi-metricd:latest "kolla_start" 20 hours ago Up 20 hours gnocchi_metricd dbe5a5e22107 192.168.24.1:8787/tripleoupstream/centos-binary-swift-object:latest "kolla_start" 20 hours ago Up 20 hours swift_object_replicator 5b527490401c 192.168.24.1:8787/tripleoupstream/centos-binary-neutron-openvswitch-agent:latest "kolla_start" 20 hours ago Up 20 hours neutronovsagent 30cca42565b5 192.168.24.1:8787/tripleoupstream/centos-binary-heat-engine:latest "kolla_start" 20 hours ago Up 20 hours heat_engine 9868fdca986b 192.168.24.1:8787/tripleoupstream/centos-binary-neutron-l3-agent:latest "kolla_start" 20 hours ago Up 20 hours neutronl3agent 7317ead2ca1a 192.168.24.1:8787/tripleoupstream/centos-binary-gnocchi-api:latest "kolla_start" 20 hours ago Up 20 hours gnocchi-api 5d785638e213 192.168.24.1:8787/tripleoupstream/centos-binary-swift-account:latest "kolla_start" 20 hours ago Up 20 hours swift_account_reaper b6f0be9a5eee 192.168.24.1:8787/tripleoupstream/centos-binary-glance-api:latest "kolla_start" 20 hours ago Up 20 hours glance_api 1aec01aff0e6 192.168.24.1:8787/tripleoupstream/centos-binary-swift-container:latest "kolla_start" 20 hours ago Up 20 hours swift_container_server 73e255c91c59 192.168.24.1:8787/tripleoupstream/centos-binary-gnocchi-statsd:latest "kolla_start" 20 hours ago Restarting (1) 5 hours ago gnocchi_statsd 4959fecd8a41 192.168.24.1:8787/tripleoupstream/centos-binary-aodh-notifier:latest "kolla_start" 20 hours ago Up 20 hours aodh_notifier a5b90ea92a14 192.168.24.1:8787/tripleoupstream/centos-binary-swift-account:latest "kolla_start" 20 hours ago Up 20 hours swift_account_replicator 5b762ed96f03 192.168.24.1:8787/tripleoupstream/centos-binary-swift-object:latest "kolla_start" 20 hours ago Up 20 hours swift_object_auditor 7edbf94532df 192.168.24.1:8787/tripleoupstream/centos-binary-heat-api:latest "kolla_start" 20 hours ago Up 20 hours heat_api d90486d0e370 192.168.24.1:8787/tripleoupstream/centos-binary-neutron-dhcp-agent:latest "kolla_start" 20 hours ago Up 20 hours neutron_dhcp abfaa4512d80 192.168.24.1:8787/tripleoupstream/centos-binary-swift-proxy-server:latest "kolla_start" 20 hours ago Up 20 hours swift_proxy 0883783d9370 192.168.24.1:8787/tripleoupstream/centos-binary-swift-object:latest "kolla_start" 20 hours ago Up 20 hours swift_object_server 8dde6c5f40c2 192.168.24.1:8787/tripleoupstream/centos-binary-nova-conductor:latest "kolla_start" 20 hours ago Up 20 hours nova_conductor 8b1e426c0295 192.168.24.1:8787/tripleoupstream/centos-binary-aodh-evaluator:latest "kolla_start" 20 hours ago Up 20 hours aodh_evaluator ee7d66308895 192.168.24.1:8787/tripleoupstream/centos-binary-swift-container:latest "kolla_start" 20 hours ago Up 20 hours swift_container_updater 6e634f364d3f 192.168.24.1:8787/tripleoupstream/centos-binary-keystone:latest "kolla_start" 20 hours ago Up 20 hours keystone 4987fb18897a 192.168.24.1:8787/tripleoupstream/centos-binary-nova-placement-api "kolla_start" 20 hours ago Up 20 hours nova_placement cf846977f5d1 192.168.24.1:8787/tripleoupstream/centos-binary-mariadb:latest "kolla_start" 20 hours ago Up 20 hours mysql b1497e32ecbc 192.168.24.1:8787/tripleoupstream/centos-binary-mongodb:latest "kolla_start" 20 hours ago Up 20 hours mongodb 77be4e8bbb61 192.168.24.1:8787/tripleoupstream/centos-binary-rabbitmq:latest "kolla_start" 20 hours ago Up 20 hours rabbitmq c480bc930940 192.168.24.1:8787/tripleoupstream/centos-binary-memcached:latest "/bin/bash -c 'source" 20 hours ago Up 20 hours memcached
Can you report the result of `docker logs gnocchi_statsd`. Docker logs will grab stdout of pid 1, which will give us a clue into what's going on.
http://pastebin.test.redhat.com/466829 docker inspect gnocchi_statsd docker logs - http://pastebin.test.redhat.com/466830
Looks like the logs were no help. What you will have to do change the run command in the container to '/bin/sleep 10000' then run '/usr/bin/gnocchi-statsd' by hand and see if we can get further output. If nothing logs to stdout, then check the log file after you try to run it. Change this command to `/bin/sleep 10000' then exec into the container: https://github.com/openstack/tripleo-heat-templates/blob/c49fa2625d27dae4dd72d67a013c0e9363d007b9/docker/services/gnocchi-statsd.yaml#L59
according to https://github.com/openstack/tripleo-heat-templates/blob/c49fa2625d27dae4dd72d67a013c0e9363d007b9/docker/services/gnocchi-statsd.yaml#L59 I have created new container #sudo docker run -ti -v /var/lib/kolla/config_files/gnocchi-statsd.json:/var/lib/kolla/config_files/config.json -v /var/lib/config-data/gnocchi:/var/lib/kolla/config_files/src 192.168.24.1:8787/tripleoupstream/centos-binary-gnocchi-statsd /bin/bash )[gnocchi@ca2468f0706b /]$ env KOLLA_CONFIG_STRATEGY=COPY_ALWAYS KOLLA_BASE_DISTRO=centos HOSTNAME=ca2468f0706b KOLLA_INSTALL_TYPE=binary TERM=xterm PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PWD=/ PS1=$(tput bold)($(printenv KOLLA_SERVICE_NAME))$(tput sgr0)[$(id -un)@$(hostname -s) $(pwd)]$ SHLVL=1 HOME=/var/lib/gnocchi KOLLA_INSTALL_METATYPE=rdo _=/usr/bin/env ()[gnocchi@ca2468f0706b /]$ kolla_start INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files INFO:__main__:Deleting file /etc/gnocchi/gnocchi.conf INFO:__main__:Coping file from /var/lib/kolla/config_files/src/etc/gnocchi/gnocchi.conf to /etc/gnocchi/gnocchi.conf INFO:__main__:Setting file /etc/gnocchi/gnocchi.conf owner to gnocchi:gnocchi INFO:__main__:Setting file /etc/gnocchi/gnocchi.conf permission to 0640 INFO:__main__:Writing out command to execute Running command: '/usr/bin/gnocchi-statsd' ()[gnocchi@ca2468f0706b /]$ echo $? 1 from logs I see keystone 404 eror http://pastebin.test.redhat.com/467126 gnocci conf ()[gnocchi@ca2468f0706b /]$ cat /etc/gnocchi/gnocchi.conf |grep -v -e '^$' |grep -v -e '^#' [DEFAULT] log_dir = /var/log/gnocchi [api] auth_mode = keystone max_limit = 1000 [archive_policy] [cors] [cors.subdomain] [database] [healthcheck] [incoming] [indexer] url = mysql+pymysql://gnocchi:p29RAUdcRT7Qfm7fGArwz6yZU.2.11/gnocchi?read_default_file=/etc/my.cnf.d/tripleo.cnf&read_default_group=tripleo [metricd] workers = 1 [oslo_middleware] enable_proxy_headers_parsing = True [oslo_policy] policy_file = /etc/gnocchi/policy.json [statsd] resource_id = 0a8b55df-f90f-491c-8cb9-7cdecec6fc26 user_id = 27c0d3f8-e7ee-42f0-8317-72237d1c5ae3 project_id = 6c38cd8d-099a-4cb2-aecf-17be688e8616 archive_policy_name = low flush_delay = 10 [storage] driver = swift swift_auth_version = 3 swift_authurl = http://172.16.2.11:5000 swift_user = service:gnocchi swift_key = p29RAUdcRT7Qfm7fGArwz6yZU coordination_url = redis://:HZYT6GPqrWwvYZxaN8QTRzV48.2.6:6379/ [keystone_authtoken] auth_uri=http://172.16.2.11:5000 auth_type=password auth_url=http://172.16.2.11:5000 username=gnocchi password=p29RAUdcRT7Qfm7fGArwz6yZU user_domain_name=Default project_name=service project_domain_name=Default ()[gnocchi@ca2468f0706b /]$ ()[gnocchi@ca2468f0706b /]$ curl -i -H "Content-Type: application/json" -d ' { "auth": { "identity": { "methods": ["password"], "password": { "user": { "name": "gnocchi", "domain": { "id": "Default" }, "password": "p29RAUdcRT7Qfm7fGArwz6yZU" } } } } }' http://172.16.2.11:5000/auth/tokens ; echo HTTP/1.1 404 Not Found Date: Wed, 22 Mar 2017 11:17:45 GMT Server: Apache Vary: X-Auth-Token Content-Length: 93 Content-Type: application/json {"error": {"message": "The resource could not be found.", "code": 404, "title": "Not Found"}} and I don't see any entries in keystone log but if I change url in request to v3 version ()[gnocchi@ca2468f0706b /]$ curl -i -H "Content-Type: application/json" -d ' { "auth": { "identity": { "methods": ["password"], "password": { "user": { "name": "gnocchi", "domain": { "name": "Default" }, "password": "p29RAUdcRT7Qfm7fGArwz6yZU" } } } } }' http://172.16.2.11:5000/v3/auth/tokens ; echo HTTP/1.1 201 Created Date: Wed, 22 Mar 2017 11:26:15 GMT Server: Apache X-Subject-Token: gAAAAABY0l9Xve1SiiMnGjU5riKy8pmi8YyZYoXsdYoFX20vwqjcJnpI0oVCo6wuaEecsM64vnOXsCBNioVP33F-M_ikC7ZbJ8HUIP1Pbus6sKNLGeEk0ZCxRKA5MmRhzSIbtiUbc9TVeENJxLeeRfWX7Okv_Iroyw Vary: X-Auth-Token x-openstack-request-id: req-e70223b7-4449-4149-a2a8-fa999a590bd6 Content-Length: 314 Content-Type: application/json {"token": {"issued_at": "2017-03-22T11:26:15.000000Z", "audit_ids": ["8xsNDdP0R3miXjfF18BDoQ"], "methods": ["password"], "expires_at": "2017-03-22T12:26:15.000000Z", "user": {"password_expires_at": null, "domain": {"id": "default", "name": "Default"}, "id": "b123ae1d06c54a6dac010885c763b61a", "name": "gnocchi"}}} and from keystone log 2017-03-22 11:25:40.027 17 INFO keystone.common.wsgi [req-2b62e610-b35d-4747-9b0c-aa8ddc425f2c - - - - -] POST http://172.16.2.11:5000/v3/auth/tokens 2017-03-22 11:25:40.130 17 WARNING keystone.common.fernet_utils [req-2b62e610-b35d-4747-9b0c-aa8ddc425f2c - - - - -] key_repository is world readable: /etc/keystone/fernet-keys Also I tryed change auth_uri and auth_url to auth_uri=http://172.16.2.11:5000/v3/ auth_url=http://172.16.2.11:5000/v3/ but result the same aking authentication request to http://172.16.2.11:5000/auth/tokens get_auth_ref /usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v3/base.py:189 2017-03-22 11:31:21.379 460 DEBUG keystoneclient.session [-] Request returned failure status: 404 request /usr/lib/python2.7/site-packages/keystoneclient/session.py:444 2017-03-22 11:31:21.379 460 CRITICAL gnocchi [-] ClientException: Authorization Failure. Authorization failed: The resource could not be found. (HTTP 404)
We're also seing this issue upstream. According to [1] the patch at https://review.openstack.org/#/c/448077/2 fixes gnocchi-statsd accessing swift but highlighted another issue where the gnocchi_statsd container image missing the redis python module. This was also fixed upstream by adding python-redis as a dependency of python-tooz but the image from https://hub.docker.com/r/tripleoupstream/centos-binary-gnocchi-statsd/ was older than the fix. I'm now uploading a newer image to the tripleoupstream registry. [1] http://logs.openstack.org/77/448077/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq-nv/a878bdf/logs/oooq/overcloud-controller-0/var/log/extra/docker/containers/gnocchi_statsd/log/gnocchi/gnocchi-statsd.log.txt.gz
The issues were fixed upstream.
unable to reproduce downstream puddle : OpenStack-12.0-RHEL-7 Puddle: 2017-10-24.3
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3457