Hide Forgot
osp-director-10 : Upgrade fails during UPGRADE CONTROLLER AND BLOCKSTORAGE, due to broken PCS cluster ( environment with no SSL ). Environment: ------------ instack-undercloud-5.0.0-2.el7ost.noarch instack-5.0.0-1.el7ost.noarch heat-cfntools-1.3.0-2.el7ost.noarch openstack-heat-api-cfn-7.0.0-4.el7ost.noarch openstack-tripleo-heat-templates-5.0.0-1.2.el7ost.noarch openstack-tripleo-heat-templates-compat-2.0.0-34.3.el7ost.noarch python-heat-agent-0-0.5.1e6015dgit.el7ost.noarch openstack-heat-api-7.0.0-4.el7ost.noarch puppet-heat-9.4.1-1.el7ost.noarch openstack-heat-templates-0-0.5.1e6015dgit.el7ost.noarch python-heatclient-1.5.0-1.el7ost.noarch openstack-heat-common-7.0.0-4.el7ost.noarch openstack-heat-engine-7.0.0-4.el7ost.noarch python-heat-tests-7.0.0-4.el7ost.noarch log can be found : ------------------- https://rhos-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/Director/view/9.0/job/infrared_deploy_9.0_3_control_1_compute_1ceph_no-UCSSL_no-OCSSL_RHEL_7.3_upgrade_10/25/console steps : -------- (1) deploy osp9 with no SSL on overcloud (2) attempt to upgrade osp9 to osp10 (3) check that pcs cluster is up and running and there are no failures before starting 'upgrade controllers and blockstorage' (4) run :upgrade controllers and blockstorage Results: ---------- upgrade fails during UPGRADE CONTROLLER AND BLOCKSTORAGE, due to broken PCS cluster ( environment with no SSL ) stack@undercloud-0 ~]$ heat deployment-show 55840ea2-07c1-4d4b-9845-8642acef7610 WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead { "status": "FAILED", "server_id": "58505421-1928-469e-a686-4699c4193d6f", "config_id": "979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92", "output_values": { "deploy_stdout": "mysql upgrade required: 0\nMon Nov 7 06:40:37 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop httpd\nMon Nov 7 06:40:39 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop memcached\nMon Nov 7 06:40:39 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop mongod\nMon Nov 7 06:40:39 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop neutron-dhcp-agent\nMon Nov 7 06:40:42 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop neutron-l3-agent\nMon Nov 7 06:40:49 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop neutron-metadata-agent\nMon Nov 7 06:40:49 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop neutron-netns-cleanup\nMon Nov 7 06:40:49 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop neutron-openvswitch-agent\nMon Nov 7 06:40:52 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop neutron-ovs-cleanup\nMon Nov 7 06:40:52 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop neutron-server\nMon Nov 7 06:41:14 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-aodh-evaluator\nMon Nov 7 06:41:14 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-aodh-listener\nMon Nov 7 06:41:15 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-aodh-notifier\nMon Nov 7 06:41:16 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-ceilometer-central\nMon Nov 7 06:41:16 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-ceilometer-collector\nMon Nov 7 06:43:05 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-ceilometer-notification\nMon Nov 7 06:43:05 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-cinder-api\nMon Nov 7 06:43:05 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-cinder-scheduler\nMon Nov 7 06:43:05 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-glance-api\nMon Nov 7 06:43:05 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-glance-registry\nMon Nov 7 06:43:06 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-gnocchi-metricd\nMon Nov 7 06:43:06 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-gnocchi-statsd\nMon Nov 7 06:43:06 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-heat-api-cfn\nMon Nov 7 06:43:06 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-heat-api\nMon Nov 7 06:43:06 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-heat-api-cloudwatch\nMon Nov 7 06:43:06 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-heat-engine\nMon Nov 7 06:43:06 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-nova-api\nMon Nov 7 06:43:06 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-nova-conductor\nMon Nov 7 06:43:06 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-nova-consoleauth\nMon Nov 7 06:43:07 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-nova-novncproxy\nMon Nov 7 06:43:07 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-nova-scheduler\nMon Nov 7 06:43:07 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-sahara-api\nMon Nov 7 06:43:07 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-sahara-engine\nMon Nov 7 06:43:08 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-account-auditor.service\nMon Nov 7 06:43:08 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-account-reaper.service\nMon Nov 7 06:43:08 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-account-replicator.service\nMon Nov 7 06:43:08 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-account.service\nMon Nov 7 06:43:08 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-container-auditor.service\nMon Nov 7 06:43:08 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-container-replicator.service\nMon Nov 7 06:43:08 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-container-updater.service\nMon Nov 7 06:43:09 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-container.service\nMon Nov 7 06:43:09 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-object-auditor.service\nMon Nov 7 06:43:09 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-object-replicator.service\nMon Nov 7 06:43:09 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-object-updater.service\nMon Nov 7 06:43:09 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-object.service\nMon Nov 7 06:43:09 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-proxy.service\nMon Nov 7 06:43:09 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-account-reaper\nMon Nov 7 06:43:10 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-account-replicator\nMon Nov 7 06:43:10 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-account\nMon Nov 7 06:43:10 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-container-auditor\nMon Nov 7 06:43:10 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-container-replicator\nMon Nov 7 06:43:10 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-container-updater\nMon Nov 7 06:43:10 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-container\nMon Nov 7 06:43:10 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-object-auditor\nMon Nov 7 06:43:10 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-object-replicator\nMon Nov 7 06:43:10 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-object-updater\nMon Nov 7 06:43:11 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-object\nMon Nov 7 06:43:11 UTC 2016 979c2b7a-fdcd-427e-b3fd-9f0db8c8ae92 tripleo-upgrade controller-2 Going to systemctl stop openstack-swift-proxy\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nactive\nERROR: cluster shutdown timed out\n", "deploy_stderr": "", "deploy_status_code": 1 }, "creation_time": "2016-11-07T06:39:16Z", "updated_time": "2016-11-07T07:13:19Z", "input_values": { "update_identifier": "", "deploy_identifier": "1478499777" }, "action": "CREATE", "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 1", "id": "55840ea2-07c1-4d4b-9845-8642acef7610" }
[stack@undercloud-0 ~]$ ssh heat-admin.2.10 Last login: Mon Nov 7 18:32:54 2016 from 192.0.2.1 [heat-admin@controller-0 ~]$ sudo su - Last login: Mon Nov 7 18:15:52 UTC 2016 from 192.0.2.1 on pts/0 [root@controller-0 ~]# pcs status Cluster name: tripleo_cluster Stack: corosync Current DC: controller-2 (version 1.1.15-11.el7-e174ec8) - partition with quorum Last updated: Mon Nov 7 19:44:24 2016 Last change: Mon Nov 7 06:01:25 2016 by root via cibadmin on controller-0 3 nodes and 124 resources configured Online: [ controller-0 controller-1 controller-2 ] Full list of resources: ip-172.17.1.10 (ocf::heartbeat:IPaddr2): Started controller-0 ip-192.0.2.6 (ocf::heartbeat:IPaddr2): Started controller-1 ip-172.17.4.10 (ocf::heartbeat:IPaddr2): Started controller-2 Clone Set: haproxy-clone [haproxy] Started: [ controller-0 controller-1 controller-2 ] Master/Slave Set: galera-master [galera] Masters: [ controller-0 controller-1 controller-2 ] Clone Set: memcached-clone [memcached] Started: [ controller-0 controller-1 controller-2 ] ip-172.17.3.10 (ocf::heartbeat:IPaddr2): Started controller-0 ip-10.0.0.101 (ocf::heartbeat:IPaddr2): Started controller-1 Clone Set: rabbitmq-clone [rabbitmq] Started: [ controller-0 controller-1 controller-2 ] Clone Set: openstack-core-clone [openstack-core] Started: [ controller-0 controller-1 controller-2 ] Master/Slave Set: redis-master [redis] Masters: [ controller-0 ] Slaves: [ controller-1 controller-2 ] ip-172.17.1.11 (ocf::heartbeat:IPaddr2): Started controller-2 Clone Set: mongod-clone [mongod] Started: [ controller-0 controller-1 controller-2 ] Clone Set: openstack-aodh-evaluator-clone [openstack-aodh-evaluator] Started: [ controller-0 controller-1 controller-2 ] Clone Set: openstack-nova-scheduler-clone [openstack-nova-scheduler] Started: [ controller-0 controller-1 controller-2 ] Clone Set: neutron-l3-agent-clone [neutron-l3-agent] Started: [ controller-0 controller-1 controller-2 ] Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup] Started: [ controller-0 controller-1 controller-2 ] Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup] Started: [ controller-0 controller-1 controller-2 ] openstack-cinder-volume (systemd:openstack-cinder-volume): Started controller-0 Clone Set: openstack-heat-engine-clone [openstack-heat-engine] Started: [ controller-0 ] Stopped: [ controller-1 controller-2 ] Clone Set: openstack-aodh-listener-clone [openstack-aodh-listener] Started: [ controller-0 controller-1 controller-2 ] Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent] Started: [ controller-0 controller-1 controller-2 ] Clone Set: openstack-gnocchi-metricd-clone [openstack-gnocchi-metricd] Started: [ controller-0 controller-1 controller-2 ] Clone Set: openstack-aodh-notifier-clone [openstack-aodh-notifier] Started: [ controller-0 controller-1 controller-2 ] Clone Set: openstack-heat-api-clone [openstack-heat-api] Started: [ controller-0 controller-1 controller-2 ] Clone Set: openstack-ceilometer-collector-clone [openstack-ceilometer-collector] Started: [ controller-0 controller-1 ] Stopped: [ controller-2 ] Clone Set: openstack-glance-api-clone [openstack-glance-api] Started: [ controller-0 controller-1 controller-2 ] Clone Set: openstack-cinder-scheduler-clone [openstack-cinder-scheduler] Started: [ controller-0 controller-1 controller-2 ] Clone Set: openstack-nova-api-clone [openstack-nova-api] Started: [ controller-0 controller-1 controller-2 ] Clone Set: openstack-nova-consoleauth-clone [openstack-nova-consoleauth] Started: [ controller-0 controller-1 controller-2 ] Clone Set: openstack-sahara-api-clone [openstack-sahara-api] Started: [ controller-0 controller-1 controller-2 ] Clone Set: openstack-heat-api-cloudwatch-clone [openstack-heat-api-cloudwatch] Started: [ controller-0 controller-1 controller-2 ] Clone Set: openstack-sahara-engine-clone [openstack-sahara-engine] Started: [ controller-0 controller-1 controller-2 ] Clone Set: openstack-glance-registry-clone [openstack-glance-registry] Started: [ controller-0 controller-1 controller-2 ] Clone Set: openstack-gnocchi-statsd-clone [openstack-gnocchi-statsd] Started: [ controller-0 controller-1 controller-2 ] Clone Set: openstack-ceilometer-notification-clone [openstack-ceilometer-notification] Started: [ controller-0 controller-1 controller-2 ] Clone Set: openstack-cinder-api-clone [openstack-cinder-api] Started: [ controller-0 controller-1 controller-2 ] Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent] Started: [ controller-0 controller-1 controller-2 ] Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent] Started: [ controller-0 controller-1 controller-2 ] Clone Set: openstack-nova-novncproxy-clone [openstack-nova-novncproxy] Started: [ controller-0 controller-1 controller-2 ] Clone Set: delay-clone [delay] Started: [ controller-0 controller-1 controller-2 ] Clone Set: neutron-server-clone [neutron-server] Started: [ controller-0 controller-1 controller-2 ] Clone Set: openstack-ceilometer-central-clone [openstack-ceilometer-central] Started: [ controller-0 controller-1 ] Stopped: [ controller-2 ] Clone Set: httpd-clone [httpd] Started: [ controller-0 controller-1 controller-2 ] Clone Set: openstack-heat-api-cfn-clone [openstack-heat-api-cfn] Started: [ controller-0 controller-1 controller-2 ] Clone Set: openstack-nova-conductor-clone [openstack-nova-conductor] Started: [ controller-0 controller-1 controller-2 ] Failed Actions: * memcached_monitor_60000 on controller-2 'not running' (7): call=41, status=complete, exitreason='none', last-rc-change='Mon Nov 7 06:41:21 2016', queued=0ms, exec=0ms * mongod_monitor_60000 on controller-2 'not running' (7): call=79, status=complete, exitreason='none', last-rc-change='Mon Nov 7 06:40:49 2016', queued=0ms, exec=0ms * openstack-aodh-evaluator_monitor_60000 on controller-2 'not running' (7): call=342, status=complete, exitreason='none', last-rc-change='Mon Nov 7 06:41:46 2016', queued=0ms, exec=0ms * neutron-l3-agent_monitor_60000 on controller-2 'not running' (7): call=120, status=complete, exitreason='none', last-rc-change='Mon Nov 7 06:41:18 2016', queued=0ms, exec=0ms * openstack-heat-engine_start_0 on controller-2 'not running' (7): call=471, status=complete, exitreason='none', last-rc-change='Mon Nov 7 06:44:07 2016', queued=0ms, exec=2280ms * openstack-aodh-listener_monitor_60000 on controller-2 'not running' (7): call=345, status=complete, exitreason='none', last-rc-change='Mon Nov 7 06:41:48 2016', queued=0ms, exec=0ms * openstack-aodh-notifier_monitor_60000 on controller-2 'not running' (7): call=346, status=complete, exitreason='none', last-rc-change='Mon Nov 7 06:41:48 2016', queued=0ms, exec=0ms * openstack-ceilometer-central_start_0 on controller-2 'not running' (7): call=439, status=complete, exitreason='none', last-rc-change='Mon Nov 7 06:43:16 2016', queued=1ms, exec=2595ms * memcached_monitor_60000 on controller-1 'not running' (7): call=45, status=complete, exitreason='none', last-rc-change='Mon Nov 7 06:41:21 2016', queued=0ms, exec=0ms * mongod_monitor_60000 on controller-1 'not running' (7): call=79, status=complete, exitreason='none', last-rc-change='Mon Nov 7 06:41:49 2016', queued=0ms, exec=0ms * openstack-aodh-evaluator_monitor_60000 on controller-1 'not running' (7): call=344, status=complete, exitreason='none', last-rc-change='Mon Nov 7 06:42:46 2016', queued=0ms, exec=0ms * neutron-l3-agent_monitor_60000 on controller-1 'OCF_PENDING' (196): call=122, status=complete, exitreason='none', last-rc-change='Mon Nov 7 06:41:19 2016', queued=0ms, exec=0ms * openstack-heat-engine_start_0 on controller-1 'not running' (7): call=466, status=complete, exitreason='none', last-rc-change='Mon Nov 7 06:44:07 2016', queued=0ms, exec=2367ms * openstack-aodh-listener_monitor_60000 on controller-1 'not running' (7): call=347, status=complete, exitreason='none', last-rc-change='Mon Nov 7 06:42:48 2016', queued=0ms, exec=0ms * openstack-aodh-notifier_monitor_60000 on controller-1 'not running' (7): call=348, status=complete, exitreason='none', last-rc-change='Mon Nov 7 06:42:48 2016', queued=0ms, exec=0ms * openstack-cinder-api_monitor_60000 on controller-1 'not running' (7): call=259, status=complete, exitreason='none', last-rc-change='Mon Nov 7 06:42:21 2016', queued=0ms, exec=0ms * openstack-ceilometer-central_monitor_60000 on controller-1 'not running' (7): call=310, status=complete, exitreason='none', last-rc-change='Mon Nov 7 06:42:30 2016', queued=0ms, exec=0ms Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled [root@controller-0 ~]#
Created attachment 1218231 [details] heat-engine.log
Created attachment 1218232 [details] messages_controller
The CI Link https://rhos-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/Director/view/9.0/job/infrared_deploy_9.0_3_control_1_compute_1ceph_no-UCSSL_no-OCSSL_RHEL_7.3_upgrade_10/25/console gives 404. Can we get sosreports or access to a live system for this one?
Deployed RHOS 9 latest Upgraded to RHOS 10 with latest puddle (2016-11-14.1) I no longer see this issue.