Bug 1324938

Summary: Please update the sensu unit files to improve behaviour when rebooting the sensu server
Product: Red Hat OpenStack Reporter: Vincent S. Cojot <vcojot>
Component: sensuAssignee: Martin Magr <mmagr>
Status: CLOSED WORKSFORME QA Contact: Leonid Natapov <lnatapov>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 11.0 (Ocata)CC: jbadiapa, lars, markmc, mburns, mmagr, mrunge, rmccabe, srevivo, vcojot
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-26 21:02:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Vincent S. Cojot 2016-04-07 16:02:53 UTC 
Description of problem:

When rebooting a sensu server, things do no restart properly.

Version-Release number of selected component (if applicable):

sensu 0.16.0 and 0.22.0 from Openstack Optools rpms..

How reproducible:

100%

Steps to Reproduce:
1. reboot.
2. sensu-server, sensu-api and sensu-client come up as dysfunctional.
3.

Actual results:


Expected results:


Additional info:

I believe the unit files of sensu-server and sensu-api should have these extra lines added:

Before=runlevel2.target runlevel3.target runlevel4.target runlevel5.target shutdown.target
After=remote-fs.target network-online.target network-online.target
Wants=network-online.target
Conflicts=shutdown.target
Comment 2 Lars Kellogg-Stedman 2017-07-17 15:38:00 UTC 
Please verify that this behavior persists under OSP 11.
Comment 3 Martin Magr 2017-08-26 21:02:12 UTC 
Tested on OSP11 deployment:

[root@overcloud-controller-0 ~]# systemctl status sensu-client
● sensu-client.service - sensu client
   Loaded: loaded (/usr/lib/systemd/system/sensu-client.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2017-08-26 20:51:27 UTC; 11s ago
 Main PID: 933112 (sensu-client)
   Memory: 17.2M
   CGroup: /system.slice/sensu-client.service
           └─933112 /usr/bin/ruby /usr/bin/sensu-client -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/sensu/extensions -l /var/log/sensu/sensu-client.log

<snip>
[root@overcloud-controller-0 ~]# tail -f /var/log/sensu/sensu-client.log
{"timestamp":"2017-08-26T20:51:33.367480+0000","level":"info","message":"received check request","check":{"command":"systemctl is-active neutron-metadata-agent || exit 2","name":"check-neutron-metadata-agent","issued":1503780693}}
{"timestamp":"2017-08-26T20:51:33.378383+0000","level":"info","message":"publishing check result","payload":{"client":"overcloud-controller-0.localdomain","check":{"command":"systemctl is-active openstack-cinder-volume || exit 2","name":"check-cinder-volume","issued":1503780693,"executed":1503780693,"duration":0.025,"output":"active\n","status":0}}}
{"timestamp":"2017-08-26T20:51:33.380163+0000","level":"info","message":"publishing check result","payload":{"client":"overcloud-controller-0.localdomain","check":{"command":"systemctl is-active pacemaker || exit 2","name":"check-pacemaker","issued":1503780693,"executed":1503780693,"duration":0.013,"output":"active\n","status":0}}}
{"timestamp":"2017-08-26T20:51:33.381132+0000","level":"info","message":"publishing check result","payload":{"client":"overcloud-controller-0.localdomain","check":{"command":"systemctl is-active neutron-metadata-agent || exit 2","name":"check-neutron-metadata-agent","issued":1503780693,"executed":1503780693,"duration":0.014,"output":"active\n","status":0}}}
{"timestamp":"2017-08-26T20:51:37.028332+0000","level":"info","message":"received check request","check":{"command":"systemctl is-active openstack-cinder-api || exit 2","name":"check-cinder-api","issued":1503780697}}
{"timestamp":"2017-08-26T20:51:37.035373+0000","level":"info","message":"publishing check result","payload":{"client":"overcloud-controller-0.localdomain","check":{"command":"systemctl is-active openstack-cinder-api || exit 2","name":"check-cinder-api","issued":1503780697,"executed":1503780697,"duration":0.007,"output":"unknown\n","status":2}}}
{"timestamp":"2017-08-26T20:51:40.840088+0000","level":"info","message":"received check request","check":{"command":"systemctl is-active memcached || exit 2","name":"check-memcached","issued":1503780700}}
{"timestamp":"2017-08-26T20:51:40.847937+0000","level":"info","message":"publishing check result","payload":{"client":"overcloud-controller-0.localdomain","check":{"command":"systemctl is-active memcached || exit 2","name":"check-memcached","issued":1503780700,"executed":1503780700,"duration":0.008,"output":"active\n","status":0}}}
{"timestamp":"2017-08-26T20:51:42.982894+0000","level":"info","message":"received check request","check":{"command":"systemctl is-active openstack-heat-api || exit 2","name":"check-heat-api","issued":1503780702}}
{"timestamp":"2017-08-26T20:51:42.988545+0000","level":"info","message":"publishing check result","payload":{"client":"overcloud-controller-0.localdomain","check":{"command":"systemctl is-active openstack-heat-api || exit 2","name":"check-heat-api","issued":1503780702,"executed":1503780702,"duration":0.005,"output":"active\n","status":0}}}
{"timestamp":"2017-08-26T20:51:47.134973+0000","level":"info","message":"received check request","check":{"command":"systemctl is-active openstack-glance-api || exit 2","name":"check-glance-api","issued":1503780707}}
{"timestamp":"2017-08-26T20:51:47.135343+0000","level":"info","message":"received check request","check":{"command":"systemctl is-active openstack-ceilometer-central || exit 2","name":"check-ceilometer-central","issued":1503780707}}
{"timestamp":"2017-08-26T20:51:47.144241+0000","level":"info","message":"publishing check result","payload":{"client":"overcloud-controller-0.localdomain","check":{"command":"systemctl is-active openstack-glance-api || exit 2","name":"check-glance-api","issued":1503780707,"executed":1503780707,"duration":0.009,"output":"active\n","status":0}}}
{"timestamp":"2017-08-26T20:51:47.145182+0000","level":"info","message":"publishing check result","payload":{"client":"overcloud-controller-0.localdomain","check":{"command":"systemctl is-active openstack-ceilometer-central || exit 2","name":"check-ceilometer-central","issued":1503780707,"executed":1503780707,"duration":0.01,"output":"active\n","status":0}}}
{"timestamp":"2017-08-26T20:51:49.497379+0000","level":"info","message":"received check request","check":{"command":"systemctl is-active openstack-heat-api-cfn || exit 2","name":"check-heat-api-cfn","issued":1503780709}}
{"timestamp":"2017-08-26T20:51:49.502629+0000","level":"info","message":"publishing check result","payload":{"client":"overcloud-controller-0.localdomain","check":{"command":"systemctl is-active openstack-heat-api-cfn || exit 2","name":"check-heat-api-cfn","issued":1503780709,"executed":1503780709,"duration":0.005,"output":"active\n","status":0}}}
:q{"timestamp":"2017-08-26T20:51:51.030378+0000","level":"info","message":"received check request","check":{"command":"systemctl is-active neutron-openvswitch-agent || exit 2","name":"check-neutron-ovs-agent","issued":1503780711}}
{"timestamp":"2017-08-26T20:51:51.036682+0000","level":"info","message":"publishing check result","payload":{"client":"overcloud-controller-0.localdomain","check":{"command":"systemctl is-active neutron-openvswitch-agent || exit 2","name":"check-neutron-ovs-agent","issued":1503780711,"executed":1503780711,"duration":0.006,"output":"active\n","status":0}}}
^C
[root@overcloud-controller-0 ~]# reboot 0
PolicyKit daemon disconnected from the bus.
We are no longer a registered authentication agent.
Connection to 192.168.24.13 closed by remote host.
Connection to 192.168.24.13 closed.
[stack@undercloud ~]$ ssh heat-admin.24.13
ssh: connect to host 192.168.24.13 port 22: Connection refused
[stack@undercloud ~]$ ssh heat-admin.24.13
ssh: connect to host 192.168.24.13 port 22: Connection refused
[stack@undercloud ~]$ ssh heat-admin.24.13
Last login: Sat Aug 26 20:46:00 2017 from 192.168.24.1
[heat-admin@overcloud-controller-0 ~]$ sudo su -
Last login: Sat Aug 26 20:49:50 UTC 2017 on pts/0
[root@overcloud-controller-0 ~]# systemctl status sensu-client
● sensu-client.service - sensu client
   Loaded: loaded (/usr/lib/systemd/system/sensu-client.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2017-08-26 20:54:14 UTC; 3min 54s ago
 Main PID: 582 (sensu-client)
   Memory: 30.1M
   CGroup: /system.slice/sensu-client.service
           └─582 /usr/bin/ruby /usr/bin/sensu-client -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/sensu/extensions -l /var/log/sensu/sensu-client.log

<snip>
[root@overcloud-controller-0 ~]# tail -f /var/log/sensu/sensu-client.log
{"timestamp":"2017-08-26T20:58:11.634974+0000","level":"info","message":"publishing check result","payload":{"client":"overcloud-controller-0.localdomain","check":{"command":"systemctl is-active openstack-nova-conductor || exit 2","name":"check-nova-conductor","issued":1503781091,"executed":1503781091,"duration":0.008,"output":"active\n","status":0}}}
{"timestamp":"2017-08-26T20:58:11.635810+0000","level":"info","message":"publishing check result","payload":{"client":"overcloud-controller-0.localdomain","check":{"command":"systemctl is-active openstack-nova-novncproxy || exit 2","name":"check-nova-novncproxy","issued":1503781091,"executed":1503781091,"duration":0.009,"output":"active\n","status":0}}}
{"timestamp":"2017-08-26T20:58:17.009447+0000","level":"info","message":"received check request","check":{"command":"systemctl is-active openstack-nova-api || exit 2","name":"check-nova-api","issued":1503781097}}
{"timestamp":"2017-08-26T20:58:17.010131+0000","level":"info","message":"received check request","check":{"command":"systemctl is-active openstack-aodh-notifier || exit 2","name":"check-aodh-notifier","issued":1503781097}}
{"timestamp":"2017-08-26T20:58:17.017813+0000","level":"info","message":"publishing check result","payload":{"client":"overcloud-controller-0.localdomain","check":{"command":"systemctl is-active openstack-nova-api || exit 2","name":"check-nova-api","issued":1503781097,"executed":1503781097,"duration":0.008,"output":"active\n","status":0}}}
{"timestamp":"2017-08-26T20:58:17.019198+0000","level":"info","message":"publishing check result","payload":{"client":"overcloud-controller-0.localdomain","check":{"command":"systemctl is-active openstack-aodh-notifier || exit 2","name":"check-aodh-notifier","issued":1503781097,"executed":1503781097,"duration":0.009,"output":"active\n","status":0}}}
{"timestamp":"2017-08-26T20:58:20.059982+0000","level":"info","message":"received check request","check":{"command":"systemctl is-active openstack-nova-scheduler || exit 2","name":"check-nova-scheduler","issued":1503781100}}
{"timestamp":"2017-08-26T20:58:20.066336+0000","level":"info","message":"publishing check result","payload":{"client":"overcloud-controller-0.localdomain","check":{"command":"systemctl is-active openstack-nova-scheduler || exit 2","name":"check-nova-scheduler","issued":1503781100,"executed":1503781100,"duration":0.006,"output":"active\n","status":0}}}
{"timestamp":"2017-08-26T20:58:24.450337+0000","level":"info","message":"received check request","check":{"command":"systemctl is-active neutron-l3-agent || exit 2","name":"check-neutron-l3-agent","issued":1503781104}}
{"timestamp":"2017-08-26T20:58:24.459353+0000","level":"info","message":"publishing check result","payload":{"client":"overcloud-controller-0.localdomain","check":{"command":"systemctl is-active neutron-l3-agent || exit 2","name":"check-neutron-l3-agent","issued":1503781104,"executed":1503781104,"duration":0.009,"output":"active\n","status":0}}}
{"timestamp":"2017-08-26T20:58:27.789231+0000","level":"info","message":"received check request","check":{"command":"/usr/bin/sudo /usr/libexec/openstack-monitoring/checks/oschecks-pacemaker_host_check -r rabbitmq-clone","name":"check-pcs-rabbitmq-clone","issued":1503781107}}
{"timestamp":"2017-08-26T20:58:29.265730+0000","level":"info","message":"publishing check result","payload":{"client":"overcloud-controller-0.localdomain","check":{"command":"/usr/bin/sudo /usr/libexec/openstack-monitoring/checks/oschecks-pacemaker_host_check -r rabbitmq-clone","name":"check-pcs-rabbitmq-clone","issued":1503781107,"executed":1503781107,"duration":1.476,"output":"\nWe trust you have received the usual lecture from the local System\nAdministrator. It usually boils down to these three things:\n\n    #1) Respect the privacy of others.\n    #2) Think before you type.\n    #3) With great power comes great responsibility.\n\nsudo: no tty present and no askpass program specified\n","status":1}}}
{"timestamp":"2017-08-26T20:58:33.350461+0000","level":"info","message":"received check request","check":{"command":"systemctl is-active openstack-cinder-volume || exit 2","name":"check-cinder-volume","issued":1503781113}}
{"timestamp":"2017-08-26T20:58:33.350717+0000","level":"info","message":"received check request","check":{"command":"systemctl is-active pacemaker || exit 2","name":"check-pacemaker","issued":1503781113}}
{"timestamp":"2017-08-26T20:58:33.350851+0000","level":"info","message":"received check request","check":{"command":"systemctl is-active neutron-metadata-agent || exit 2","name":"check-neutron-metadata-agent","issued":1503781113}}
{"timestamp":"2017-08-26T20:58:33.360378+0000","level":"info","message":"publishing check result","payload":{"client":"overcloud-controller-0.localdomain","check":{"command":"systemctl is-active neutron-metadata-agent || exit 2","name":"check-neutron-metadata-agent","issued":1503781113,"executed":1503781113,"duration":0.009,"output":"active\n","status":0}}}
{"timestamp":"2017-08-26T20:58:33.360881+0000","level":"info","message":"publishing check result","payload":{"client":"overcloud-controller-0.localdomain","check":{"command":"systemctl is-active pacemaker || exit 2","name":"check-pacemaker","issued":1503781113,"executed":1503781113,"duration":0.01,"output":"active\n","status":0}}}
{"timestamp":"2017-08-26T20:58:33.361206+0000","level":"info","message":"publishing check result","payload":{"client":"overcloud-controller-0.localdomain","check":{"command":"systemctl is-active openstack-cinder-volume || exit 2","name":"check-cinder-volume","issued":1503781113,"executed":1503781113,"duration":0.011,"output":"active\n","status":0}}}
{"timestamp":"2017-08-26T20:58:37.031674+0000","level":"info","message":"received check request","check":{"command":"systemctl is-active openstack-cinder-api || exit 2","name":"check-cinder-api","issued":1503781117}}
{"timestamp":"2017-08-26T20:58:37.036963+0000","level":"info","message":"publishing check result","payload":{"client":"overcloud-controller-0.localdomain","check":{"command":"systemctl is-active openstack-cinder-api || exit 2","name":"check-cinder-api","issued":1503781117,"executed":1503781117,"duration":0.005,"output":"unknown\n","status":2}}}
^C
[root@overcloud-controller-0 ~]#