Bug 1392579

Summary:

osp-director-10 : Upgrade fails during UPGRADE CONTROLLER AND BLOCKSTORAGE, due to broken PCS cluster ( environment with no SSL ).

Product:

Red Hat OpenStack

Reporter:

Omri Hochman <ohochman>

Component:

rhosp-director

Assignee:

Michele Baldessari <michele>

Status:

CLOSED CURRENTRELEASE

QA Contact:

Omri Hochman <ohochman>

Severity:

urgent

Docs Contact:

Priority:

urgent

Version:

10.0 (Newton)

CC:

dbecker, jcoufal, mandreou, mburns, mcornea, michele, mlammon, morazi, ohochman, rhel-osp-director-maint

Target Milestone:

Keywords:

Triaged

Target Release:

10.0 (Newton)

Hardware:

x86_64

OS:

Linux

Whiteboard:

Fixed In Version:

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2016-12-16 16:50:51 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Attachments:

Description	Flags
heat-engine.log	none
messages_controller	none

Description Omri Hochman 2016-11-07 19:47:06 UTC

osp-director-10 : Upgrade fails during UPGRADE CONTROLLER AND BLOCKSTORAGE, due to broken PCS cluster  ( environment with no SSL ).


Environment: 
------------
instack-undercloud-5.0.0-2.el7ost.noarch
instack-5.0.0-1.el7ost.noarch
heat-cfntools-1.3.0-2.el7ost.noarch
openstack-heat-api-cfn-7.0.0-4.el7ost.noarch
openstack-tripleo-heat-templates-5.0.0-1.2.el7ost.noarch
openstack-tripleo-heat-templates-compat-2.0.0-34.3.el7ost.noarch
python-heat-agent-0-0.5.1e6015dgit.el7ost.noarch
openstack-heat-api-7.0.0-4.el7ost.noarch
puppet-heat-9.4.1-1.el7ost.noarch
openstack-heat-templates-0-0.5.1e6015dgit.el7ost.noarch
python-heatclient-1.5.0-1.el7ost.noarch
openstack-heat-common-7.0.0-4.el7ost.noarch
openstack-heat-engine-7.0.0-4.el7ost.noarch
python-heat-tests-7.0.0-4.el7ost.noarch



log can be found :
-------------------
https://rhos-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/Director/view/9.0/job/infrared_deploy_9.0_3_control_1_compute_1ceph_no-UCSSL_no-OCSSL_RHEL_7.3_upgrade_10/25/console

steps :
--------
(1) deploy osp9 with no SSL on overcloud
(2) attempt to upgrade osp9 to osp10 
(3) check that pcs cluster is up and running and there are no failures before starting 'upgrade controllers and blockstorage'
(4) run :upgrade controllers and blockstorage


Results:   
----------
upgrade fails during UPGRADE CONTROLLER AND BLOCKSTORAGE, due to broken PCS cluster  ( environment with no SSL ) 


stack@undercloud-0 ~]$ heat deployment-show 55840ea2-07c1-4d4b-9845-8642acef7610
WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead
{
  "status": "FAILED", 
  "server_id": "58505421-1928-469e-a686-4699c4193d6f", 
  "config_id": "979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92", 
  "output_values": {
    "deploy_stdout": "mysql upgrade required: 0\nMon Nov  7 06:40:37 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop httpd\nMon Nov  7 06:40:39 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop memcached\nMon Nov  7 06:40:39 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop mongod\nMon Nov  7 06:40:39 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop neutron-dhcp-agent\nMon Nov  7 06:40:42 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop neutron-l3-agent\nMon Nov  7 06:40:49 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop neutron-metadata-agent\nMon Nov  7 06:40:49 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop neutron-netns-cleanup\nMon Nov  7 06:40:49 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop neutron-openvswitch-agent\nMon Nov  7 06:40:52 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop neutron-ovs-cleanup\nMon Nov  7 06:40:52 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop neutron-server\nMon Nov  7 06:41:14 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-aodh-evaluator\nMon Nov  7 06:41:14 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-aodh-listener\nMon Nov  7 06:41:15 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-aodh-notifier\nMon Nov  7 06:41:16 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-ceilometer-central\nMon Nov  7 06:41:16 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-ceilometer-collector\nMon Nov  7 06:43:05 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-ceilometer-notification\nMon Nov  7 06:43:05 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-cinder-api\nMon Nov  7 06:43:05 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-cinder-scheduler\nMon Nov  7 06:43:05 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-glance-api\nMon Nov  7 06:43:05 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-glance-registry\nMon Nov  7 06:43:06 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-gnocchi-metricd\nMon Nov  7 06:43:06 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-gnocchi-statsd\nMon Nov  7 06:43:06 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-heat-api-cfn\nMon Nov  7 06:43:06 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-heat-api\nMon Nov  7 06:43:06 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-heat-api-cloudwatch\nMon Nov  7 06:43:06 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-heat-engine\nMon Nov  7 06:43:06 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-nova-api\nMon Nov  7 06:43:06 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-nova-conductor\nMon Nov  7 06:43:06 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-nova-consoleauth\nMon Nov  7 06:43:07 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-nova-novncproxy\nMon Nov  7 06:43:07 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-nova-scheduler\nMon Nov  7 06:43:07 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-sahara-api\nMon Nov  7 06:43:07 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-sahara-engine\nMon Nov  7 06:43:08 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-account-auditor.service\nMon Nov  7 06:43:08 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-account-reaper.service\nMon Nov  7 06:43:08 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-account-replicator.service\nMon Nov  7 06:43:08 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-account.service\nMon Nov  7 06:43:08 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-container-auditor.service\nMon Nov  7 06:43:08 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-container-replicator.service\nMon Nov  7 06:43:08 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-container-updater.service\nMon Nov  7 06:43:09 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-container.service\nMon Nov  7 06:43:09 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-object-auditor.service\nMon Nov  7 06:43:09 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-object-replicator.service\nMon Nov  7 06:43:09 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-object-updater.service\nMon Nov  7 06:43:09 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-object.service\nMon Nov  7 06:43:09 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-proxy.service\nMon Nov  7 06:43:09 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-account-reaper\nMon Nov  7 06:43:10 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-account-replicator\nMon Nov  7 06:43:10 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-account\nMon Nov  7 06:43:10 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-container-auditor\nMon Nov  7 06:43:10 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-container-replicator\nMon Nov  7 06:43:10 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-container-updater\nMon Nov  7 06:43:10 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-container\nMon Nov  7 06:43:10 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-object-auditor\nMon Nov  7 06:43:10 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-object-replicator\nMon Nov  7 06:43:10 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-object-updater\nMon Nov  7 06:43:11 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-object\nMon Nov  7 06:43:11 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-proxy\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nERROR: cluster shutdown timed out\n", 
    "deploy_stderr": "", 
    "deploy_status_code": 1
  }, 
  "creation_time": "2016-11-07T06:39:16Z", 
  "updated_time": "2016-11-07T07:13:19Z", 
  "input_values": {
    "update_identifier": "", 
    "deploy_identifier": "1478499777"
  }, 
  "action": "CREATE", 
  "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 1", 
  "id": "55840ea2-07c1-4d4b-9845-8642acef7610"
}

Comment 1 Omri Hochman 2016-11-07 19:47:23 UTC

[stack@undercloud-0 ~]$ ssh heat-admin.2.10
Last login: Mon Nov  7 18:32:54 2016 from 192.0.2.1
[heat-admin@controller-0 ~]$ sudo su -
Last login: Mon Nov  7 18:15:52 UTC 2016 from 192.0.2.1 on pts/0
[root@controller-0 ~]# pcs status
Cluster name: tripleo_cluster
Stack: corosync
Current DC: controller-2 (version 1.1.15-11.el7-e174ec8) - partition with quorum
Last updated: Mon Nov  7 19:44:24 2016          Last change: Mon Nov  7 06:01:25 2016 by root via cibadmin on controller-0

3 nodes and 124 resources configured

Online: [ controller-0 controller-1 controller-2 ]

Full list of resources:

 ip-172.17.1.10 (ocf::heartbeat:IPaddr2):       Started controller-0
 ip-192.0.2.6   (ocf::heartbeat:IPaddr2):       Started controller-1
 ip-172.17.4.10 (ocf::heartbeat:IPaddr2):       Started controller-2
 Clone Set: haproxy-clone [haproxy]
     Started: [ controller-0 controller-1 controller-2 ]
 Master/Slave Set: galera-master [galera]
     Masters: [ controller-0 controller-1 controller-2 ]
 Clone Set: memcached-clone [memcached]
     Started: [ controller-0 controller-1 controller-2 ]
 ip-172.17.3.10 (ocf::heartbeat:IPaddr2):       Started controller-0
 ip-10.0.0.101  (ocf::heartbeat:IPaddr2):       Started controller-1
 Clone Set: rabbitmq-clone [rabbitmq]
     Started: [ controller-0 controller-1 controller-2 ]
 Clone Set: openstack-core-clone [openstack-core]
     Started: [ controller-0 controller-1 controller-2 ]
 Master/Slave Set: redis-master [redis]
     Masters: [ controller-0 ]
     Slaves: [ controller-1 controller-2 ]
 ip-172.17.1.11 (ocf::heartbeat:IPaddr2):       Started controller-2
 Clone Set: mongod-clone [mongod]
     Started: [ controller-0 controller-1 controller-2 ]
 Clone Set: openstack-aodh-evaluator-clone [openstack-aodh-evaluator]
     Started: [ controller-0 controller-1 controller-2 ]
 Clone Set: openstack-nova-scheduler-clone [openstack-nova-scheduler]
     Started: [ controller-0 controller-1 controller-2 ]
 Clone Set: neutron-l3-agent-clone [neutron-l3-agent]
     Started: [ controller-0 controller-1 controller-2 ]
 Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup]
     Started: [ controller-0 controller-1 controller-2 ]
 Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup]
     Started: [ controller-0 controller-1 controller-2 ]
 openstack-cinder-volume        (systemd:openstack-cinder-volume):      Started controller-0
 Clone Set: openstack-heat-engine-clone [openstack-heat-engine]
     Started: [ controller-0 ]
     Stopped: [ controller-1 controller-2 ]
 Clone Set: openstack-aodh-listener-clone [openstack-aodh-listener]
     Started: [ controller-0 controller-1 controller-2 ]
 Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent]
     Started: [ controller-0 controller-1 controller-2 ]
 Clone Set: openstack-gnocchi-metricd-clone [openstack-gnocchi-metricd]
     Started: [ controller-0 controller-1 controller-2 ]
 Clone Set: openstack-aodh-notifier-clone [openstack-aodh-notifier]
     Started: [ controller-0 controller-1 controller-2 ]
 Clone Set: openstack-heat-api-clone [openstack-heat-api]
     Started: [ controller-0 controller-1 controller-2 ]
 Clone Set: openstack-ceilometer-collector-clone [openstack-ceilometer-collector]
     Started: [ controller-0 controller-1 ]
     Stopped: [ controller-2 ]
 Clone Set: openstack-glance-api-clone [openstack-glance-api]
     Started: [ controller-0 controller-1 controller-2 ]
 Clone Set: openstack-cinder-scheduler-clone [openstack-cinder-scheduler]
     Started: [ controller-0 controller-1 controller-2 ]
 Clone Set: openstack-nova-api-clone [openstack-nova-api]
     Started: [ controller-0 controller-1 controller-2 ]
 Clone Set: openstack-nova-consoleauth-clone [openstack-nova-consoleauth]
     Started: [ controller-0 controller-1 controller-2 ]
 Clone Set: openstack-sahara-api-clone [openstack-sahara-api]
     Started: [ controller-0 controller-1 controller-2 ]
 Clone Set: openstack-heat-api-cloudwatch-clone [openstack-heat-api-cloudwatch]
     Started: [ controller-0 controller-1 controller-2 ]
 Clone Set: openstack-sahara-engine-clone [openstack-sahara-engine]
     Started: [ controller-0 controller-1 controller-2 ]
 Clone Set: openstack-glance-registry-clone [openstack-glance-registry]
     Started: [ controller-0 controller-1 controller-2 ]
 Clone Set: openstack-gnocchi-statsd-clone [openstack-gnocchi-statsd]
     Started: [ controller-0 controller-1 controller-2 ]
 Clone Set: openstack-ceilometer-notification-clone [openstack-ceilometer-notification]
     Started: [ controller-0 controller-1 controller-2 ]
 Clone Set: openstack-cinder-api-clone [openstack-cinder-api]
     Started: [ controller-0 controller-1 controller-2 ]
 Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent]
     Started: [ controller-0 controller-1 controller-2 ]
 Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent]
     Started: [ controller-0 controller-1 controller-2 ]
 Clone Set: openstack-nova-novncproxy-clone [openstack-nova-novncproxy]
     Started: [ controller-0 controller-1 controller-2 ]
 Clone Set: delay-clone [delay]
     Started: [ controller-0 controller-1 controller-2 ]
 Clone Set: neutron-server-clone [neutron-server]
     Started: [ controller-0 controller-1 controller-2 ]
 Clone Set: openstack-ceilometer-central-clone [openstack-ceilometer-central]
     Started: [ controller-0 controller-1 ]
     Stopped: [ controller-2 ]
 Clone Set: httpd-clone [httpd]
     Started: [ controller-0 controller-1 controller-2 ]
 Clone Set: openstack-heat-api-cfn-clone [openstack-heat-api-cfn]
     Started: [ controller-0 controller-1 controller-2 ]
 Clone Set: openstack-nova-conductor-clone [openstack-nova-conductor]
     Started: [ controller-0 controller-1 controller-2 ]

Failed Actions:
* memcached_monitor_60000 on controller-2 'not running' (7): call=41, status=complete, exitreason='none',
    last-rc-change='Mon Nov  7 06:41:21 2016', queued=0ms, exec=0ms
* mongod_monitor_60000 on controller-2 'not running' (7): call=79, status=complete, exitreason='none',
    last-rc-change='Mon Nov  7 06:40:49 2016', queued=0ms, exec=0ms
* openstack-aodh-evaluator_monitor_60000 on controller-2 'not running' (7): call=342, status=complete, exitreason='none',
    last-rc-change='Mon Nov  7 06:41:46 2016', queued=0ms, exec=0ms
* neutron-l3-agent_monitor_60000 on controller-2 'not running' (7): call=120, status=complete, exitreason='none',
    last-rc-change='Mon Nov  7 06:41:18 2016', queued=0ms, exec=0ms
* openstack-heat-engine_start_0 on controller-2 'not running' (7): call=471, status=complete, exitreason='none',
    last-rc-change='Mon Nov  7 06:44:07 2016', queued=0ms, exec=2280ms
* openstack-aodh-listener_monitor_60000 on controller-2 'not running' (7): call=345, status=complete, exitreason='none',
    last-rc-change='Mon Nov  7 06:41:48 2016', queued=0ms, exec=0ms
* openstack-aodh-notifier_monitor_60000 on controller-2 'not running' (7): call=346, status=complete, exitreason='none',
    last-rc-change='Mon Nov  7 06:41:48 2016', queued=0ms, exec=0ms
* openstack-ceilometer-central_start_0 on controller-2 'not running' (7): call=439, status=complete, exitreason='none',
    last-rc-change='Mon Nov  7 06:43:16 2016', queued=1ms, exec=2595ms
* memcached_monitor_60000 on controller-1 'not running' (7): call=45, status=complete, exitreason='none',
    last-rc-change='Mon Nov  7 06:41:21 2016', queued=0ms, exec=0ms
* mongod_monitor_60000 on controller-1 'not running' (7): call=79, status=complete, exitreason='none',
    last-rc-change='Mon Nov  7 06:41:49 2016', queued=0ms, exec=0ms
* openstack-aodh-evaluator_monitor_60000 on controller-1 'not running' (7): call=344, status=complete, exitreason='none',
    last-rc-change='Mon Nov  7 06:42:46 2016', queued=0ms, exec=0ms
* neutron-l3-agent_monitor_60000 on controller-1 'OCF_PENDING' (196): call=122, status=complete, exitreason='none',
    last-rc-change='Mon Nov  7 06:41:19 2016', queued=0ms, exec=0ms
* openstack-heat-engine_start_0 on controller-1 'not running' (7): call=466, status=complete, exitreason='none',
    last-rc-change='Mon Nov  7 06:44:07 2016', queued=0ms, exec=2367ms
* openstack-aodh-listener_monitor_60000 on controller-1 'not running' (7): call=347, status=complete, exitreason='none',
    last-rc-change='Mon Nov  7 06:42:48 2016', queued=0ms, exec=0ms
* openstack-aodh-notifier_monitor_60000 on controller-1 'not running' (7): call=348, status=complete, exitreason='none',
    last-rc-change='Mon Nov  7 06:42:48 2016', queued=0ms, exec=0ms
* openstack-cinder-api_monitor_60000 on controller-1 'not running' (7): call=259, status=complete, exitreason='none',
    last-rc-change='Mon Nov  7 06:42:21 2016', queued=0ms, exec=0ms
* openstack-ceilometer-central_monitor_60000 on controller-1 'not running' (7): call=310, status=complete, exitreason='none',
    last-rc-change='Mon Nov  7 06:42:30 2016', queued=0ms, exec=0ms


Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
[root@controller-0 ~]#

Comment 3 Omri Hochman 2016-11-07 19:55:12 UTC

Created attachment 1218231 [details]
heat-engine.log

Comment 4 Omri Hochman 2016-11-07 19:55:36 UTC

Created attachment 1218232 [details]
messages_controller

Comment 6 Michele Baldessari 2016-11-08 16:21:47 UTC

The CI Link https://rhos-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/Director/view/9.0/job/infrared_deploy_9.0_3_control_1_compute_1ceph_no-UCSSL_no-OCSSL_RHEL_7.3_upgrade_10/25/console gives 404.

Can we get sosreports or access to a live system for this one?

Comment 9 mlammon 2016-11-15 18:51:44 UTC

Deployed RHOS 9 latest
Upgraded to RHOS 10 with latest puddle (2016-11-14.1)

I no longer see this issue.