Hide Forgot
osp-director-10: Upgrade OSP9 - OSP10 leaves PCS Failure Actions on controllers for openstack-gnocchi-metricd_start (service is down after the init step) Environment ( undercloud) : --------------------------- instack-undercloud-5.0.0-0.20160818065636.41ef775.el7ost.noarch instack-5.0.0-0.20160802165724.5aabf5c.el7ost.noarch openstack-heat-templates-0.0.1-0.20160822094546.1ac2823.el7ost.noarch openstack-tripleo-heat-templates-liberty-2.0.0-33.el7ost.noarch openstack-heat-engine-7.0.0-0.20160823082523.1106458.el7ost.noarch openstack-heat-api-7.0.0-0.20160823082523.1106458.el7ost.noarch openstack-tripleo-heat-templates-5.0.0-0.20160823140311.72404b.1.el7ost.noarch heat-cfntools-1.3.0-2.el7ost.noarch openstack-heat-common-7.0.0-0.20160823082523.1106458.el7ost.noarch openstack-heat-api-cfn-7.0.0-0.20160823082523.1106458.el7ost.noarch puppet-heat-9.1.0-0.20160815142726.d364553.el7ost.noarch python-heatclient-1.3.0-0.20160802194627.44dfe53.el7ost.noarch python-heat-tests-7.0.0-0.20160823082523.1106458.el7ost.noarch environment: (overcloud-controllers) ------------------------------------- python-gnocchiclient-2.2.0-1.el7ost.noarch openstack-gnocchi-indexer-sqlalchemy-2.1.3-3.el7ost.noarch openstack-gnocchi-statsd-2.1.3-3.el7ost.noarch openstack-gnocchi-common-2.1.3-3.el7ost.noarch openstack-gnocchi-metricd-2.1.3-3.el7ost.noarch python-gnocchi-2.1.3-3.el7ost.noarch openstack-gnocchi-api-2.1.3-3.el7ost.noarch openstack-gnocchi-carbonara-2.1.3-3.el7ost.noarch Steps: ------- (1) Attempt to upgrade OSP9 to OSP10 using https://gitlab.cee.redhat.com/sathlang/ospd-9-to-10-upgrade#controller-and-block-storage-upgrade (2) After the init step the PCS Cluster is down, reported in https://bugzilla.redhat.com/show_bug.cgi?id=1372829 (3) After starting the PCS cluster it seems that there are failed resources for openstack-gnocchi-metricd_start [heat-admin@overcloud-controller-1 ~]$ sudo pcs status Cluster name: tripleo_cluster Last updated: Thu Sep 8 23:52:27 2016 Last change: Thu Sep 8 19:03:19 2016 by root via cibadmin on overcloud-controller-0 Stack: corosync Current DC: overcloud-controller-2 (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum 3 nodes and 127 resources configured Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Full list of resources: ip-10.19.184.210 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0 ip-192.168.200.10 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1 Clone Set: haproxy-clone [haproxy] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] ip-192.168.0.6 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2 Master/Slave Set: galera-master [galera] Masters: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] ip-10.19.104.11 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0 Clone Set: memcached-clone [memcached] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] ip-10.19.105.10 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1 Clone Set: rabbitmq-clone [rabbitmq] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-core-clone [openstack-core] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Master/Slave Set: redis-master [redis] Masters: [ overcloud-controller-2 ] Slaves: [ overcloud-controller-0 overcloud-controller-1 ] ip-10.19.104.10 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2 Clone Set: mongod-clone [mongod] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-aodh-evaluator-clone [openstack-aodh-evaluator] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-nova-scheduler-clone [openstack-nova-scheduler] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: neutron-l3-agent-clone [neutron-l3-agent] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] openstack-cinder-volume (systemd:openstack-cinder-volume): Started overcloud-controller-0 Clone Set: openstack-heat-engine-clone [openstack-heat-engine] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-ceilometer-api-clone [openstack-ceilometer-api] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-aodh-listener-clone [openstack-aodh-listener] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-gnocchi-metricd-clone [openstack-gnocchi-metricd] Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-aodh-notifier-clone [openstack-aodh-notifier] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-heat-api-clone [openstack-heat-api] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-ceilometer-collector-clone [openstack-ceilometer-collector] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-glance-api-clone [openstack-glance-api] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-cinder-scheduler-clone [openstack-cinder-scheduler] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-nova-api-clone [openstack-nova-api] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-nova-consoleauth-clone [openstack-nova-consoleauth] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-sahara-api-clone [openstack-sahara-api] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-heat-api-cloudwatch-clone [openstack-heat-api-cloudwatch] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-sahara-engine-clone [openstack-sahara-engine] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-glance-registry-clone [openstack-glance-registry] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-gnocchi-statsd-clone [openstack-gnocchi-statsd] Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-ceilometer-notification-clone [openstack-ceilometer-notification] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-cinder-api-clone [openstack-cinder-api] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-nova-novncproxy-clone [openstack-nova-novncproxy] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: delay-clone [delay] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: neutron-server-clone [neutron-server] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-ceilometer-central-clone [openstack-ceilometer-central] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: httpd-clone [httpd] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-heat-api-cfn-clone [openstack-heat-api-cfn] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-nova-conductor-clone [openstack-nova-conductor] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Failed Actions: * openstack-gnocchi-metricd_start_0 on overcloud-controller-2 'not running' (7): call=324, status=complete, exitreason='none', last-rc-change='Thu Sep 8 18:58:03 2016', queued=0ms, exec=2216ms * openstack-gnocchi-statsd_start_0 on overcloud-controller-2 'not running' (7): call=237, status=complete, exitreason='none', last-rc-change='Thu Sep 8 18:56:55 2016', queued=0ms, exec=2759ms * openstack-gnocchi-metricd_start_0 on overcloud-controller-0 'not running' (7): call=328, status=complete, exitreason='none', last-rc-change='Thu Sep 8 18:58:03 2016', queued=0ms, exec=2226ms * openstack-gnocchi-statsd_start_0 on overcloud-controller-0 'not running' (7): call=242, status=complete, exitreason='none', last-rc-change='Thu Sep 8 18:56:55 2016', queued=0ms, exec=2504ms * openstack-gnocchi-metricd_start_0 on overcloud-controller-1 'not running' (7): call=326, status=complete, exitreason='none', last-rc-change='Thu Sep 8 18:58:03 2016', queued=0ms, exec=2216ms * openstack-gnocchi-statsd_start_0 on overcloud-controller-1 'not running' (7): call=239, status=complete, exitreason='none', last-rc-change='Thu Sep 8 18:56:55 2016', queued=0ms, exec=2777ms PCSD Status: overcloud-controller-0: Online overcloud-controller-1: Online overcloud-controller-2: Online Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled Note : Attempt to start the service manually worked. ------------------------------------------------------- [heat-admin@overcloud-controller-1 ~]$ sudo systemctl status openstack-gnocchi-statsd.service ● openstack-gnocchi-statsd.service - OpenStack gnocchi statsd service Loaded: loaded (/usr/lib/systemd/system/openstack-gnocchi-statsd.service; disabled; vendor preset: disabled) Active: failed (Result: exit-code) since Thu 2016-09-08 18:56:58 UTC; 5h 6min ago Main PID: 8578 (code=exited, status=1/FAILURE) Sep 08 18:56:56 overcloud-controller-1.localdomain systemd[1]: Started Cluster Controlled openstack-gnocchi-statsd. Sep 08 18:56:56 overcloud-controller-1.localdomain systemd[1]: Starting Cluster Controlled openstack-gnocchi-statsd... Sep 08 18:56:58 overcloud-controller-1.localdomain systemd[1]: openstack-gnocchi-statsd.service: main process exited, code=exited, status=1/FAILURE Sep 08 18:56:58 overcloud-controller-1.localdomain systemd[1]: Unit openstack-gnocchi-statsd.service entered failed state. Sep 08 18:56:58 overcloud-controller-1.localdomain systemd[1]: openstack-gnocchi-statsd.service failed. [heat-admin@overcloud-controller-1 ~]$ sudo systemctl start openstack-gnocchi-statsd.service [heat-admin@overcloud-controller-1 ~]$ sudo systemctl status openstack-gnocchi-statsd.service ● openstack-gnocchi-statsd.service - OpenStack gnocchi statsd service Loaded: loaded (/usr/lib/systemd/system/openstack-gnocchi-statsd.service; disabled; vendor preset: disabled) Active: active (running) since Fri 2016-09-09 00:03:37 UTC; 356ms ago Main PID: 14131 (gnocchi-statsd) CGroup: /system.slice/openstack-gnocchi-statsd.service └─14131 /usr/bin/python2 /usr/bin/gnocchi-statsd --logfile /var/log/gnocchi/statsd.log Sep 09 00:03:37 overcloud-controller-1.localdomain systemd[1]: Started OpenStack gnocchi statsd service. Sep 09 00:03:37 overcloud-controller-1.localdomain systemd[1]: Starting OpenStack gnocchi statsd service... [heat-admin@overcloud-controller-1 ~]$
closing this as not a bug based on the comments at https://bugzilla.redhat.com/show_bug.cgi?id=1372829#c5 please reopen if you disagree
closed, no need for needinfo.