Bug 1374531

Summary: osp-director-10: Upgrade OSP9 - OSP10 leaves PCS Failure Actions on controllers for openstack-gnocchi-metricd_start (service is down after the init step)
Product: Red Hat OpenStack Reporter: Omri Hochman <ohochman>
Component: rhosp-directorAssignee: Angus Thomas <athomas>
Status: CLOSED NOTABUG QA Contact: Omri Hochman <ohochman>
Severity: medium Docs Contact:
Priority: medium    
Version: 10.0 (Newton)CC: augol, dbecker, jcoufal, mandreou, mburns, morazi, ohochman, rhel-osp-director-maint
Target Milestone: ---Keywords: Triaged
Target Release: 10.0 (Newton)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-10-13 10:39:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Omri Hochman 2016-09-09 00:13:00 UTC 
osp-director-10:  Upgrade OSP9 - OSP10 leaves PCS Failure Actions on controllers for openstack-gnocchi-metricd_start  (service is down after the init step)

Environment ( undercloud) :
---------------------------
instack-undercloud-5.0.0-0.20160818065636.41ef775.el7ost.noarch
instack-5.0.0-0.20160802165724.5aabf5c.el7ost.noarch
openstack-heat-templates-0.0.1-0.20160822094546.1ac2823.el7ost.noarch
openstack-tripleo-heat-templates-liberty-2.0.0-33.el7ost.noarch
openstack-heat-engine-7.0.0-0.20160823082523.1106458.el7ost.noarch
openstack-heat-api-7.0.0-0.20160823082523.1106458.el7ost.noarch
openstack-tripleo-heat-templates-5.0.0-0.20160823140311.72404b.1.el7ost.noarch
heat-cfntools-1.3.0-2.el7ost.noarch
openstack-heat-common-7.0.0-0.20160823082523.1106458.el7ost.noarch
openstack-heat-api-cfn-7.0.0-0.20160823082523.1106458.el7ost.noarch
puppet-heat-9.1.0-0.20160815142726.d364553.el7ost.noarch
python-heatclient-1.3.0-0.20160802194627.44dfe53.el7ost.noarch
python-heat-tests-7.0.0-0.20160823082523.1106458.el7ost.noarch

environment: (overcloud-controllers) 
-------------------------------------
python-gnocchiclient-2.2.0-1.el7ost.noarch
openstack-gnocchi-indexer-sqlalchemy-2.1.3-3.el7ost.noarch
openstack-gnocchi-statsd-2.1.3-3.el7ost.noarch
openstack-gnocchi-common-2.1.3-3.el7ost.noarch
openstack-gnocchi-metricd-2.1.3-3.el7ost.noarch
python-gnocchi-2.1.3-3.el7ost.noarch
openstack-gnocchi-api-2.1.3-3.el7ost.noarch
openstack-gnocchi-carbonara-2.1.3-3.el7ost.noarch


Steps: 
-------

(1) Attempt to upgrade OSP9 to OSP10 using https://gitlab.cee.redhat.com/sathlang/ospd-9-to-10-upgrade#controller-and-block-storage-upgrade

(2) After the init step the PCS Cluster is down, reported in https://bugzilla.redhat.com/show_bug.cgi?id=1372829


(3) After starting the PCS cluster it seems that there are failed resources for 
openstack-gnocchi-metricd_start 


[heat-admin@overcloud-controller-1 ~]$ sudo pcs status
Cluster name: tripleo_cluster
Last updated: Thu Sep  8 23:52:27 2016          Last change: Thu Sep  8 19:03:19 2016 by root via cibadmin on overcloud-controller-0
Stack: corosync
Current DC: overcloud-controller-2 (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum
3 nodes and 127 resources configured

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Full list of resources:

 ip-10.19.184.210       (ocf::heartbeat:IPaddr2):       Started overcloud-controller-0
 ip-192.168.200.10      (ocf::heartbeat:IPaddr2):       Started overcloud-controller-1
 Clone Set: haproxy-clone [haproxy]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 ip-192.168.0.6 (ocf::heartbeat:IPaddr2):       Started overcloud-controller-2
 Master/Slave Set: galera-master [galera]
     Masters: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 ip-10.19.104.11        (ocf::heartbeat:IPaddr2):       Started overcloud-controller-0
 Clone Set: memcached-clone [memcached]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 ip-10.19.105.10        (ocf::heartbeat:IPaddr2):       Started overcloud-controller-1
 Clone Set: rabbitmq-clone [rabbitmq]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-core-clone [openstack-core]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Master/Slave Set: redis-master [redis]
     Masters: [ overcloud-controller-2 ]
     Slaves: [ overcloud-controller-0 overcloud-controller-1 ]
 ip-10.19.104.10        (ocf::heartbeat:IPaddr2):       Started overcloud-controller-2
 Clone Set: mongod-clone [mongod]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-evaluator-clone [openstack-aodh-evaluator]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-scheduler-clone [openstack-nova-scheduler]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-l3-agent-clone [neutron-l3-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 openstack-cinder-volume        (systemd:openstack-cinder-volume):      Started overcloud-controller-0
 Clone Set: openstack-heat-engine-clone [openstack-heat-engine]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-api-clone [openstack-ceilometer-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-listener-clone [openstack-aodh-listener]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-gnocchi-metricd-clone [openstack-gnocchi-metricd]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-notifier-clone [openstack-aodh-notifier]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-clone [openstack-heat-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-collector-clone [openstack-ceilometer-collector]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-glance-api-clone [openstack-glance-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-cinder-scheduler-clone [openstack-cinder-scheduler]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-api-clone [openstack-nova-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-consoleauth-clone [openstack-nova-consoleauth]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-sahara-api-clone [openstack-sahara-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-cloudwatch-clone [openstack-heat-api-cloudwatch]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-sahara-engine-clone [openstack-sahara-engine]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-glance-registry-clone [openstack-glance-registry]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-gnocchi-statsd-clone [openstack-gnocchi-statsd]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-notification-clone [openstack-ceilometer-notification]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-cinder-api-clone [openstack-cinder-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-novncproxy-clone [openstack-nova-novncproxy]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: delay-clone [delay]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-server-clone [neutron-server]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-central-clone [openstack-ceilometer-central]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: httpd-clone [httpd]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-cfn-clone [openstack-heat-api-cfn]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-conductor-clone [openstack-nova-conductor]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Failed Actions:
* openstack-gnocchi-metricd_start_0 on overcloud-controller-2 'not running' (7): call=324, status=complete, exitreason='none',
    last-rc-change='Thu Sep  8 18:58:03 2016', queued=0ms, exec=2216ms
* openstack-gnocchi-statsd_start_0 on overcloud-controller-2 'not running' (7): call=237, status=complete, exitreason='none',
    last-rc-change='Thu Sep  8 18:56:55 2016', queued=0ms, exec=2759ms
* openstack-gnocchi-metricd_start_0 on overcloud-controller-0 'not running' (7): call=328, status=complete, exitreason='none',
    last-rc-change='Thu Sep  8 18:58:03 2016', queued=0ms, exec=2226ms
* openstack-gnocchi-statsd_start_0 on overcloud-controller-0 'not running' (7): call=242, status=complete, exitreason='none',
    last-rc-change='Thu Sep  8 18:56:55 2016', queued=0ms, exec=2504ms
* openstack-gnocchi-metricd_start_0 on overcloud-controller-1 'not running' (7): call=326, status=complete, exitreason='none',
    last-rc-change='Thu Sep  8 18:58:03 2016', queued=0ms, exec=2216ms
* openstack-gnocchi-statsd_start_0 on overcloud-controller-1 'not running' (7): call=239, status=complete, exitreason='none',
    last-rc-change='Thu Sep  8 18:56:55 2016', queued=0ms, exec=2777ms


PCSD Status:
  overcloud-controller-0: Online
  overcloud-controller-1: Online
  overcloud-controller-2: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled



Note : Attempt to start the service manually worked.  
-------------------------------------------------------

[heat-admin@overcloud-controller-1 ~]$ sudo systemctl status openstack-gnocchi-statsd.service
● openstack-gnocchi-statsd.service - OpenStack gnocchi statsd service
   Loaded: loaded (/usr/lib/systemd/system/openstack-gnocchi-statsd.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2016-09-08 18:56:58 UTC; 5h 6min ago
 Main PID: 8578 (code=exited, status=1/FAILURE)

Sep 08 18:56:56 overcloud-controller-1.localdomain systemd[1]: Started Cluster Controlled openstack-gnocchi-statsd.
Sep 08 18:56:56 overcloud-controller-1.localdomain systemd[1]: Starting Cluster Controlled openstack-gnocchi-statsd...
Sep 08 18:56:58 overcloud-controller-1.localdomain systemd[1]: openstack-gnocchi-statsd.service: main process exited, code=exited, status=1/FAILURE
Sep 08 18:56:58 overcloud-controller-1.localdomain systemd[1]: Unit openstack-gnocchi-statsd.service entered failed state.
Sep 08 18:56:58 overcloud-controller-1.localdomain systemd[1]: openstack-gnocchi-statsd.service failed.



[heat-admin@overcloud-controller-1 ~]$ sudo systemctl start openstack-gnocchi-statsd.service



[heat-admin@overcloud-controller-1 ~]$ sudo systemctl status openstack-gnocchi-statsd.service
● openstack-gnocchi-statsd.service - OpenStack gnocchi statsd service
   Loaded: loaded (/usr/lib/systemd/system/openstack-gnocchi-statsd.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2016-09-09 00:03:37 UTC; 356ms ago
 Main PID: 14131 (gnocchi-statsd)
   CGroup: /system.slice/openstack-gnocchi-statsd.service
           └─14131 /usr/bin/python2 /usr/bin/gnocchi-statsd --logfile /var/log/gnocchi/statsd.log

Sep 09 00:03:37 overcloud-controller-1.localdomain systemd[1]: Started OpenStack gnocchi statsd service.
Sep 09 00:03:37 overcloud-controller-1.localdomain systemd[1]: Starting OpenStack gnocchi statsd service...

[heat-admin@overcloud-controller-1 ~]$
Comment 2 Marios Andreou 2016-10-13 10:39:02 UTC 
closing this as not a bug based on the comments at https://bugzilla.redhat.com/show_bug.cgi?id=1372829#c5 please reopen if you disagree
Comment 4 Amit Ugol 2018-05-02 10:54:16 UTC 
closed, no need for needinfo.