OSP15 | sensu | containers health check reports "Failed to connect to bus" instead of reporting health status of containers. The report is seen in Uchiwa. When executing container health check script manually (not inside the container) it works fine. The script should be executed inside each container fetching data from host. It seems like the problem is that for some reason on rhel8 the container does not have access to host.
For some reason sensu-client does not see systemd data from within container on RHEL8. There has to be some differences from RHEL7/CentOS7. Gonna investigate: [root@controller-0 ~]# podman exec -it sensu_client bash ()[root@controller-0 /]# systemctl list-timers --no-pager --no-legend "tripleo*healthcheck.timer" Failed to connect to bus: No data available
Possibly connected selinux avc: ---- time->Thu Jul 11 12:14:13 2019 type=PROCTITLE msg=audit(1562847253.125:10112): proctitle=73797374656D63746C006C6973742D74696D657273002D2D6E6F2D7061676572002D2D6E6F2D6C6567656E6400747269706C656F2A6865616C7468636865636B2E74696D6572 type=SYSCALL msg=audit(1562847253.125:10112): arch=c000003e syscall=42 success=yes exit=0 a0=3 a1=561dee599f70 a2=16 a3=7ffc35113670 items=0 ppid=602403 pid=602452 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=3 comm="systemctl" exe="/usr/bin/systemctl" subj=system_u:system_r:container_t:s0:c181,c206 key=(null) type=AVC msg=audit(1562847253.125:10112): avc: denied { connectto } for pid=602452 comm="systemctl" path="/run/systemd/private" scontext=system_u:system_r:container_t:s0:c181,c206 tcontext=system_u:system_r:init_t:s0 tclass=unix_stream_socket permissive=1
One thing we will need to fix is AVC denials in SELinux: [root@controller-0 ~]# podman run --network=host --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/dev/log:/dev/log --volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro --volume=/etc/puppet:/etc/puppet:ro --volume=/var/log/journal:/var/log/journal:ro --volume=/sys/fs/cgroup:/sys/fs/cgroup --volume=/run:/run --volume=/usr/lib/systemd:/usr/lib/systemd --volume=/var/lib/kolla/config_files/sensu-client.json:/var/lib/kolla/config_files/config.json:ro --volume=/var/lib/config-data/puppet-generated/sensu/:/var/lib/kolla/config_files/src:ro --volume=/var/log/containers/sensu:/var/log/sensu:rw,z 7138c84aec57 systemctl list-timers --no-pager --no-legend "tripleo*healthcheck.timer" Failed to connect to bus: Permission denied [root@controller-0 ~]# ausearch -m avc <snip> ---- time->Tue Aug 6 12:20:42 2019 type=PROCTITLE msg=audit(1565094042.901:130455): proctitle=73797374656D63746C006C6973742D74696D657273002D2D6E6F2D7061676572002D2D6E6F2D6C6567656E6400747269706C656F2A6865616C7468636865636B2E74696D6572 type=SYSCALL msg=audit(1565094042.901:130455): arch=c000003e syscall=42 success=no exit=-13 a0=3 a1=55974d280730 a2=1d a3=7fffa95761d0 items=0 ppid=812841 pid=812858 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=60 comm="systemctl" exe="/usr/bin/systemctl" subj=system_u:system_r:container_t:s0:c400,c976 key=(null) type=AVC msg=audit(1565094042.901:130455): avc: denied { connectto } for pid=812858 comm="systemctl" path="/run/dbus/system_bus_socket" scontext=system_u:system_r:container_t:s0:c400,c976 tcontext=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 tclass=unix_stream_socket permissive=0 [root@controller-0 ~]# setenforce 0 [root@controller-0 ~]# podman run --network=host --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/dev/log:/dev/log --volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro --volume=/etc/puppet:/etc/puppet:ro --volume=/var/log/journal:/var/log/journal:ro --volume=/sys/fs/cgroup:/sys/fs/cgroup --volume=/run:/run --volume=/usr/lib/systemd:/usr/lib/systemd --volume=/var/lib/kolla/config_files/sensu-client.json:/var/lib/kolla/config_files/config.json:ro --volume=/var/lib/config-data/puppet-generated/sensu/:/var/lib/kolla/config_files/src:ro --volume=/var/log/containers/sensu:/var/log/sensu:rw,z 7138c84aec57 systemctl list-timers --no-pager --no-legend "tripleo*healthcheck.timer" Failed to connect to bus: No data available [root@controller-0 ~]# The next thing is that we gonna need to add systemd-udev package to sensu-client kolla image: [root@controller-0 ~]# podman run --systemd --network=host --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/dev/log:/dev/log --volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro --volume=/etc/puppet:/etc/puppet:ro --volume=/var/log/journal:/var/log/journal:ro --volume=/sys/fs/cgroup:/sys/fs/cgroup --volume=/run/dbus/system_bus_socket://run/dbus/system_bus_socket:rw,z --volume=/run:/run --volume=/usr/lib/systemd:/usr/lib/systemd:rw,z --volume=/var/lib/kolla/config_files/sensu-client.json:/var/lib/kolla/config_files/config.json:ro --volume=/var/lib/config-data/puppet-generated/sensu/:/var/lib/kolla/config_files/src:ro --volume=/var/log/containers/sensu:/var/log/sensu:rw,z 7138c84aec57 yum install -y systemd-udev && systemctl list-timers --no-pager --no-legend "tripleo*healthcheck.timer" relabel failed "/usr/lib/systemd": relabeling content in /usr is not allowed [root@controller-0 ~]# podman run --systemd --network=host --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/dev/log:/dev/log --volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro --volume=/etc/puppet:/etc/puppet:ro --volume=/var/log/journal:/var/log/journal:ro --volume=/sys/fs/cgroup:/sys/fs/cgroup --volume=/run/dbus/system_bus_socket://run/dbus/system_bus_socket:rw,z --volume=/run:/run --volume=/usr/lib/systemd:/usr/lib/systemd:rw --volume=/var/lib/kolla/config_files/sensu-client.json:/var/lib/kolla/config_files/config.json:ro --volume=/var/lib/config-data/puppet-generated/sensu/:/var/lib/kolla/config_files/src:ro --volume=/var/log/containers/sensu:/var/log/sensu:rw,z 7138c84aec57 yum install -y systemd-udev && systemctl list-timers --no-pager --no-legend "tripleo*healthcheck.timer" Updating Subscription Management repositories. Unable to read consumer identity Subscription Manager is operating in container mode. This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. Red Hat Universal Base Image 8 (RPMs) - AppStre 550 kB/s | 1.9 MB 00:03 Red Hat Universal Base Image 8 (RPMs) - BaseOS 224 kB/s | 737 kB 00:03 Dependencies resolved. ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: systemd-udev x86_64 239-13.el8_0.5 ubi-8-baseos 1.3 M Transaction Summary ================================================================================ Install 1 Package Total download size: 1.3 M Installed size: 7.8 M Downloading Packages: systemd-udev-239-13.el8_0.5.x86_64.rpm 670 kB/s | 1.3 MB 00:01 -------------------------------------------------------------------------------- Total 669 kB/s | 1.3 MB 00:01 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : systemd-udev-239-13.el8_0.5.x86_64 1/1 Running scriptlet: systemd-udev-239-13.el8_0.5.x86_64 1/1 Failed to connect to bus: No data available warning: %triggerin(systemd-239-13.el8_0.5.x86_64) scriptlet failed, exit status 1 Error in <unknown> scriptlet in rpm package systemd-udev Verifying : systemd-udev-239-13.el8_0.5.x86_64 1/1 Installed products updated. Installed: systemd-udev-239-13.el8_0.5.x86_64 Complete! Tue 2019-08-06 12:57:13 UTC 58ms left Tue 2019-08-06 12:55:44 UTC 1min 28s ago tripleo_collectd_healthcheck.timer tripleo_collectd_healthcheck.service Tue 2019-08-06 12:57:13 UTC 170ms left Tue 2019-08-06 12:56:11 UTC 1min 2s ago tripleo_heat_engine_healthcheck.timer tripleo_heat_engine_healthcheck.service Tue 2019-08-06 12:57:15 UTC 1s left Tue 2019-08-06 12:56:11 UTC 1min 2s ago tripleo_heat_api_healthcheck.timer tripleo_heat_api_healthcheck.service Tue 2019-08-06 12:57:15 UTC 1s left Tue 2019-08-06 12:55:54 UTC 1min 18s ago tripleo_swift_object_server_healthcheck.timer tripleo_swift_object_server_healthcheck.service Tue 2019-08-06 12:57:16 UTC 3s left Tue 2019-08-06 12:56:11 UTC 1min 2s ago tripleo_nova_metadata_healthcheck.timer tripleo_nova_metadata_healthcheck.service Tue 2019-08-06 12:57:24 UTC 10s left Tue 2019-08-06 12:55:45 UTC 1min 28s ago tripleo_glance_api_healthcheck.timer tripleo_glance_api_healthcheck.service Tue 2019-08-06 12:57:27 UTC 14s left Tue 2019-08-06 12:56:15 UTC 57s ago tripleo_aodh_api_healthcheck.timer tripleo_aodh_api_healthcheck.service Tue 2019-08-06 12:57:27 UTC 14s left Tue 2019-08-06 12:55:51 UTC 1min 22s ago tripleo_keystone_healthcheck.timer tripleo_keystone_healthcheck.service Tue 2019-08-06 12:57:27 UTC 14s left Tue 2019-08-06 12:56:01 UTC 1min 12s ago tripleo_memcached_healthcheck.timer tripleo_memcached_healthcheck.service Tue 2019-08-06 12:57:28 UTC 15s left Tue 2019-08-06 12:55:44 UTC 1min 28s ago tripleo_aodh_evaluator_healthcheck.timer tripleo_aodh_evaluator_healthcheck.service Tue 2019-08-06 12:57:29 UTC 16s left Tue 2019-08-06 12:56:01 UTC 1min 12s ago tripleo_swift_container_server_healthcheck.timer tripleo_swift_container_server_healthcheck.service Tue 2019-08-06 12:57:32 UTC 18s left Tue 2019-08-06 12:56:01 UTC 1min 12s ago tripleo_cinder_api_healthcheck.timer tripleo_cinder_api_healthcheck.service Tue 2019-08-06 12:57:34 UTC 21s left Tue 2019-08-06 12:56:31 UTC 42s ago tripleo_nova_placement_healthcheck.timer tripleo_nova_placement_healthcheck.service Tue 2019-08-06 12:57:36 UTC 22s left Tue 2019-08-06 12:56:31 UTC 42s ago tripleo_ovn_controller_healthcheck.timer tripleo_ovn_controller_healthcheck.service Tue 2019-08-06 12:57:39 UTC 26s left Tue 2019-08-06 12:56:31 UTC 42s ago tripleo_heat_api_cron_healthcheck.timer tripleo_heat_api_cron_healthcheck.service Tue 2019-08-06 12:57:40 UTC 27s left Tue 2019-08-06 12:56:11 UTC 1min 2s ago tripleo_cinder_scheduler_healthcheck.timer tripleo_cinder_scheduler_healthcheck.service Tue 2019-08-06 12:57:42 UTC 29s left Tue 2019-08-06 12:56:15 UTC 57s ago tripleo_nova_consoleauth_healthcheck.timer tripleo_nova_consoleauth_healthcheck.service Tue 2019-08-06 12:57:42 UTC 29s left Tue 2019-08-06 12:56:35 UTC 37s ago tripleo_heat_api_cfn_healthcheck.timer tripleo_heat_api_cfn_healthcheck.service Tue 2019-08-06 12:57:47 UTC 34s left Tue 2019-08-06 12:56:41 UTC 32s ago tripleo_swift_account_server_healthcheck.timer tripleo_swift_account_server_healthcheck.service Tue 2019-08-06 12:57:47 UTC 34s left Tue 2019-08-06 12:56:11 UTC 1min 2s ago tripleo_logrotate_crond_healthcheck.timer tripleo_logrotate_crond_healthcheck.service Tue 2019-08-06 12:57:50 UTC 36s left Tue 2019-08-06 12:56:21 UTC 52s ago tripleo_iscsid_healthcheck.timer tripleo_iscsid_healthcheck.service Tue 2019-08-06 12:57:50 UTC 37s left Tue 2019-08-06 12:56:35 UTC 38s ago tripleo_nova_vnc_proxy_healthcheck.timer tripleo_nova_vnc_proxy_healthcheck.service Tue 2019-08-06 12:57:52 UTC 38s left Tue 2019-08-06 12:56:31 UTC 42s ago tripleo_neutron_api_healthcheck.timer tripleo_neutron_api_healthcheck.service Tue 2019-08-06 12:57:55 UTC 42s left Tue 2019-08-06 12:56:11 UTC 1min 2s ago tripleo_sensu_client_healthcheck.timer tripleo_sensu_client_healthcheck.service Tue 2019-08-06 12:57:56 UTC 43s left Tue 2019-08-06 12:56:44 UTC 29s ago tripleo_ceilometer_agent_notification_healthcheck.timer tripleo_ceilometer_agent_notification_healthcheck.service Tue 2019-08-06 12:57:56 UTC 43s left Tue 2019-08-06 12:56:31 UTC 42s ago tripleo_gnocchi_statsd_healthcheck.timer tripleo_gnocchi_statsd_healthcheck.service Tue 2019-08-06 12:57:58 UTC 45s left Tue 2019-08-06 12:56:41 UTC 32s ago tripleo_keystone_cron_healthcheck.timer tripleo_keystone_cron_healthcheck.service Tue 2019-08-06 12:58:01 UTC 47s left Tue 2019-08-06 12:56:51 UTC 22s ago tripleo_gnocchi_api_healthcheck.timer tripleo_gnocchi_api_healthcheck.service Tue 2019-08-06 12:58:03 UTC 49s left Tue 2019-08-06 12:57:01 UTC 12s ago tripleo_nova_scheduler_healthcheck.timer tripleo_nova_scheduler_healthcheck.service Tue 2019-08-06 12:58:06 UTC 53s left Tue 2019-08-06 12:56:31 UTC 42s ago tripleo_aodh_notifier_healthcheck.timer tripleo_aodh_notifier_healthcheck.service Tue 2019-08-06 12:58:09 UTC 56s left Tue 2019-08-06 12:57:06 UTC 6s ago tripleo_swift_proxy_healthcheck.timer tripleo_swift_proxy_healthcheck.service Tue 2019-08-06 12:58:12 UTC 59s left Tue 2019-08-06 12:56:35 UTC 38s ago tripleo_nova_api_cron_healthcheck.timer tripleo_nova_api_cron_healthcheck.service Tue 2019-08-06 12:58:14 UTC 1min 1s left Tue 2019-08-06 12:56:31 UTC 41s ago tripleo_gnocchi_metricd_healthcheck.timer tripleo_gnocchi_metricd_healthcheck.service Tue 2019-08-06 12:58:14 UTC 1min 1s left Tue 2019-08-06 12:56:45 UTC 27s ago tripleo_nova_conductor_healthcheck.timer tripleo_nova_conductor_healthcheck.service Tue 2019-08-06 12:58:15 UTC 1min 2s left Tue 2019-08-06 12:57:06 UTC 6s ago tripleo_cinder_api_cron_healthcheck.timer tripleo_cinder_api_cron_healthcheck.service Tue 2019-08-06 12:58:19 UTC 1min 6s left Tue 2019-08-06 12:57:06 UTC 6s ago tripleo_panko_api_healthcheck.timer tripleo_panko_api_healthcheck.service Tue 2019-08-06 12:58:21 UTC 1min 7s left Tue 2019-08-06 12:56:41 UTC 32s ago tripleo_ceilometer_agent_central_healthcheck.timer tripleo_ceilometer_agent_central_healthcheck.service Tue 2019-08-06 12:58:26 UTC 1min 12s left Tue 2019-08-06 12:56:51 UTC 22s ago tripleo_aodh_listener_healthcheck.timer tripleo_aodh_listener_healthcheck.service Tue 2019-08-06 12:58:33 UTC 1min 20s left Tue 2019-08-06 12:57:01 UTC 12s ago tripleo_fluentd_healthcheck.timer tripleo_fluentd_healthcheck.service Tue 2019-08-06 12:58:44 UTC 1min 30s left Tue 2019-08-06 12:57:11 UTC 2s ago tripleo_nova_api_healthcheck.timer tripleo_nova_api_healthcheck.service [root@controller-0 ~]#
So with systemd-udev installed the container still behaves incorrectly: [root@controller-0 ~]# podman run --systemd --network=host --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/dev/log:/dev/log --volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro --volume=/etc/puppet:/etc/puppet:ro --volume=/var/log/journal:/var/log/journal:ro --volume=/sys/fs/cgroup:/sys/fs/cgroup --volume=/run/dbus/system_bus_socket://run/dbus/system_bus_socket:rw,z --volume=/run:/run --volume=/usr/lib/systemd:/usr/lib/systemd:rw --volume=/var/lib/kolla/config_files/sensu-client.json:/var/lib/kolla/config_files/config.json:ro --volume=/var/lib/config-data/puppet-generated/sensu/:/var/lib/kolla/config_files/src:ro --volume=/var/log/containers/sensu:/var/log/sensu:rw,z 2cc104aac809 yum install -y systemd-udev && systemctl list-timers --no-pager --no-legend "tripleo*healthcheck.timer" Updating Subscription Management repositories. Unable to read consumer identity Subscription Manager is operating in container mode. could not create lock This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. Red Hat Universal Base Image 8 (RPMs) - AppStre 1.0 MB/s | 1.9 MB 00:01 Red Hat Universal Base Image 8 (RPMs) - BaseOS 388 kB/s | 745 kB 00:01 Last metadata expiration check: 0:00:01 ago on Wed Aug 28 11:10:57 2019. Package systemd-udev-239-13.el8_0.5.x86_64 is already installed. Dependencies resolved. Nothing to do. Complete! Wed 2019-08-28 11:10:58 UTC 49ms left Wed 2019-08-28 11:09:54 UTC 1min 4s ago tripleo_keystone_cron_healthcheck.timer tripleo_keystone_cron_healthcheck.service Wed 2019-08-28 11:11:00 UTC 1s left Wed 2019-08-28 11:09:54 UTC 1min 4s ago tripleo_ceilometer_agent_central_healthcheck.timer tripleo_ceilometer_agent_central_healthcheck.service Wed 2019-08-28 11:11:01 UTC 3s left Wed 2019-08-28 11:09:54 UTC 1min 4s ago tripleo_panko_api_healthcheck.timer tripleo_panko_api_healthcheck.service Wed 2019-08-28 11:11:04 UTC 5s left Wed 2019-08-28 11:09:56 UTC 1min 2s ago tripleo_aodh_evaluator_healthcheck.timer tripleo_aodh_evaluator_healthcheck.service Wed 2019-08-28 11:11:07 UTC 8s left Wed 2019-08-28 11:09:54 UTC 1min 4s ago tripleo_gnocchi_metricd_healthcheck.timer tripleo_gnocchi_metricd_healthcheck.service Wed 2019-08-28 11:11:08 UTC 9s left Wed 2019-08-28 11:09:54 UTC 1min 4s ago tripleo_nova_placement_healthcheck.timer tripleo_nova_placement_healthcheck.service Wed 2019-08-28 11:11:08 UTC 9s left Wed 2019-08-28 11:09:54 UTC 1min 4s ago tripleo_swift_object_server_healthcheck.timer tripleo_swift_object_server_healthcheck.service Wed 2019-08-28 11:11:09 UTC 10s left Wed 2019-08-28 11:09:54 UTC 1min 4s ago tripleo_nova_conductor_healthcheck.timer tripleo_nova_conductor_healthcheck.service Wed 2019-08-28 11:11:13 UTC 14s left Wed 2019-08-28 11:09:54 UTC 1min 4s ago tripleo_memcached_healthcheck.timer tripleo_memcached_healthcheck.service Wed 2019-08-28 11:11:15 UTC 16s left Wed 2019-08-28 11:09:54 UTC 1min 4s ago tripleo_cinder_api_healthcheck.timer tripleo_cinder_api_healthcheck.service Wed 2019-08-28 11:11:15 UTC 16s left Wed 2019-08-28 11:09:54 UTC 1min 4s ago tripleo_cinder_scheduler_healthcheck.timer tripleo_cinder_scheduler_healthcheck.service Wed 2019-08-28 11:11:15 UTC 16s left Wed 2019-08-28 11:09:54 UTC 1min 4s ago tripleo_sensu_client_healthcheck.timer tripleo_sensu_client_healthcheck.service Wed 2019-08-28 11:11:17 UTC 18s left Wed 2019-08-28 11:09:54 UTC 1min 4s ago tripleo_ovn_controller_healthcheck.timer tripleo_ovn_controller_healthcheck.service Wed 2019-08-28 11:11:20 UTC 21s left Wed 2019-08-28 11:09:54 UTC 1min 4s ago tripleo_swift_account_server_healthcheck.timer tripleo_swift_account_server_healthcheck.service Wed 2019-08-28 11:11:20 UTC 21s left Wed 2019-08-28 11:09:54 UTC 1min 4s ago tripleo_nova_scheduler_healthcheck.timer tripleo_nova_scheduler_healthcheck.service Wed 2019-08-28 11:11:20 UTC 21s left Wed 2019-08-28 11:09:54 UTC 1min 4s ago tripleo_swift_container_server_healthcheck.timer tripleo_swift_container_server_healthcheck.service Wed 2019-08-28 11:11:20 UTC 21s left Wed 2019-08-28 11:09:54 UTC 1min 4s ago tripleo_aodh_api_healthcheck.timer tripleo_aodh_api_healthcheck.service Wed 2019-08-28 11:11:23 UTC 24s left Wed 2019-08-28 11:09:54 UTC 1min 4s ago tripleo_aodh_listener_healthcheck.timer tripleo_aodh_listener_healthcheck.service Wed 2019-08-28 11:11:23 UTC 24s left Wed 2019-08-28 11:09:54 UTC 1min 4s ago tripleo_nova_api_cron_healthcheck.timer tripleo_nova_api_cron_healthcheck.service Wed 2019-08-28 11:11:26 UTC 27s left Wed 2019-08-28 11:09:54 UTC 1min 4s ago tripleo_glance_api_healthcheck.timer tripleo_glance_api_healthcheck.service Wed 2019-08-28 11:11:27 UTC 28s left Wed 2019-08-28 11:09:54 UTC 1min 4s ago tripleo_heat_engine_healthcheck.timer tripleo_heat_engine_healthcheck.service Wed 2019-08-28 11:11:33 UTC 34s left Wed 2019-08-28 11:09:54 UTC 1min 4s ago tripleo_iscsid_healthcheck.timer tripleo_iscsid_healthcheck.service Wed 2019-08-28 11:11:34 UTC 35s left Wed 2019-08-28 11:09:54 UTC 1min 4s ago tripleo_nova_metadata_healthcheck.timer tripleo_nova_metadata_healthcheck.service Wed 2019-08-28 11:11:34 UTC 35s left Wed 2019-08-28 11:09:54 UTC 1min 4s ago tripleo_ceilometer_agent_notification_healthcheck.timer tripleo_ceilometer_agent_notification_healthcheck.service Wed 2019-08-28 11:11:34 UTC 35s left Wed 2019-08-28 11:09:54 UTC 1min 4s ago tripleo_heat_api_healthcheck.timer tripleo_heat_api_healthcheck.service Wed 2019-08-28 11:11:35 UTC 37s left Wed 2019-08-28 11:09:54 UTC 1min 4s ago tripleo_nova_vnc_proxy_healthcheck.timer tripleo_nova_vnc_proxy_healthcheck.service Wed 2019-08-28 11:11:38 UTC 39s left Wed 2019-08-28 11:09:54 UTC 1min 4s ago tripleo_nova_api_healthcheck.timer tripleo_nova_api_healthcheck.service Wed 2019-08-28 11:11:38 UTC 39s left Wed 2019-08-28 11:09:54 UTC 1min 4s ago tripleo_heat_api_cron_healthcheck.timer tripleo_heat_api_cron_healthcheck.service Wed 2019-08-28 11:11:53 UTC 54s left Wed 2019-08-28 11:10:51 UTC 7s ago tripleo_swift_proxy_healthcheck.timer tripleo_swift_proxy_healthcheck.service Wed 2019-08-28 11:12:01 UTC 1min 2s left Wed 2019-08-28 11:10:51 UTC 7s ago tripleo_cinder_api_cron_healthcheck.timer tripleo_cinder_api_cron_healthcheck.service Wed 2019-08-28 11:12:02 UTC 1min 3s left Wed 2019-08-28 11:10:51 UTC 7s ago tripleo_logrotate_crond_healthcheck.timer tripleo_logrotate_crond_healthcheck.service Wed 2019-08-28 11:12:08 UTC 1min 9s left Wed 2019-08-28 11:10:51 UTC 7s ago tripleo_neutron_api_healthcheck.timer tripleo_neutron_api_healthcheck.service Wed 2019-08-28 11:12:12 UTC 1min 13s left Wed 2019-08-28 11:10:51 UTC 7s ago tripleo_gnocchi_statsd_healthcheck.timer tripleo_gnocchi_statsd_healthcheck.service Wed 2019-08-28 11:12:18 UTC 1min 19s left Wed 2019-08-28 11:10:51 UTC 7s ago tripleo_heat_api_cfn_healthcheck.timer tripleo_heat_api_cfn_healthcheck.service Wed 2019-08-28 11:12:23 UTC 1min 24s left Wed 2019-08-28 11:10:51 UTC 7s ago tripleo_aodh_notifier_healthcheck.timer tripleo_aodh_notifier_healthcheck.service Wed 2019-08-28 11:12:24 UTC 1min 25s left Wed 2019-08-28 11:10:51 UTC 7s ago tripleo_nova_consoleauth_healthcheck.timer tripleo_nova_consoleauth_healthcheck.service Wed 2019-08-28 11:12:31 UTC 1min 32s left Wed 2019-08-28 11:10:51 UTC 7s ago tripleo_gnocchi_api_healthcheck.timer tripleo_gnocchi_api_healthcheck.service Wed 2019-08-28 11:12:36 UTC 1min 37s left Wed 2019-08-28 11:10:51 UTC 7s ago tripleo_keystone_healthcheck.timer tripleo_keystone_healthcheck.service [root@controller-0 ~]# podman run --systemd --network=host --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/dev/log:/dev/log --volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro --volume=/etc/puppet:/etc/puppet:ro --volume=/var/log/journal:/var/log/journal:ro --volume=/sys/fs/cgroup:/sys/fs/cgroup --volume=/run/dbus/system_bus_socket://run/dbus/system_bus_socket:rw,z --volume=/run:/run --volume=/usr/lib/systemd:/usr/lib/systemd:rw --volume=/var/lib/kolla/config_files/sensu-client.json:/var/lib/kolla/config_files/config.json:ro --volume=/var/lib/config-data/puppet-generated/sensu/:/var/lib/kolla/config_files/src:ro --volume=/var/log/containers/sensu:/var/log/sensu:rw,z 2cc104aac809 systemctl list-timers --no-pager --no-legend "tripleo*healthcheck.timer" Failed to connect to bus: No data available [root@controller-0 ~]# TripleO deployed container also does not have proper systemd access: [root@controller-0 ~]# podman exec -it sensu_client bash ()[root@controller-0 /]# systemctl list-timers --no-pager --no-legend "tripleo*healthcheck.timer" Failed to connect to bus: No data available ()[root@controller-0 /]# yum install -y systemd-udev Updating Subscription Management repositories. Unable to read consumer identity Subscription Manager is operating in container mode. This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. Last metadata expiration check: 0:00:15 ago on Wed Aug 28 11:18:43 2019. Package systemd-udev-239-13.el8_0.5.x86_64 is already installed. Dependencies resolved. Nothing to do. Complete! ()[root@controller-0 /]# systemctl list-timers --no-pager --no-legend "tripleo*healthcheck.timer" Failed to connect to bus: No data available ()[root@controller-0 /]# This BZ FailedQA.
Martin, can you get AVCs from a permissive run? That should tell us what we need to know.
Never mind - Julie pointed me at the other bug.
The question in https://bugzilla.redhat.com/show_bug.cgi?id=1728226#c15 was answered already. Clearing needinfo.
Ok I found the cause. When container is started with --pid=host then the behaviour is correct: [root@controller-0 ~]# podman run --systemd --pid=host --network=host --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/dev/log:/dev/log --volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro --volume=/etc/puppet:/etc/puppet:ro --volume=/var/log/journal:/var/log/journal:ro --volume=/sys/fs/cgroup:/sys/fs/cgroup --volume=/run/dbus/system_bus_socket://run/dbus/system_bus_socket:rw,z --volume=/run:/run --volume=/usr/lib/systemd:/usr/lib/systemd:rw --volume=/var/lib/kolla/config_files/sensu-client.json:/var/lib/kolla/config_files/config.json:ro --volume=/var/lib/config-data/puppet-generated/sensu/:/var/lib/kolla/config_files/src:ro --volume=/var/log/containers/sensu:/var/log/sensu:rw,z d3ad6556ad74 systemctl list-timers --no-pager --no-legend "tripleo*healthcheck.timer" Tue 2019-09-03 11:19:39 UTC 3s left Tue 2019-09-03 11:17:55 UTC 1min 39s ago tripleo_logrotate_crond_healthcheck.timer tripleo_logrotate_crond_healthcheck.service Tue 2019-09-03 11:19:42 UTC 6s left Tue 2019-09-03 11:18:10 UTC 1min 25s ago tripleo_ceilometer_agent_notification_healthcheck.timer tripleo_ceilometer_agent_notification_healthcheck.service Tue 2019-09-03 11:19:44 UTC 8s left Tue 2019-09-03 11:18:37 UTC 58s ago tripleo_nova_placement_healthcheck.timer tripleo_nova_placement_healthcheck.service Tue 2019-09-03 11:19:44 UTC 9s left Tue 2019-09-03 11:18:35 UTC 59s ago tripleo_glance_api_healthcheck.timer tripleo_glance_api_healthcheck.service We just need to make TripleO to run sensu-client container with that option now.
Failed QA. The initial problem is gone but we have another issue now that causes the same result e.g. container health report is not working properly. Moving back to Assign.
Now container health check reports: "Running in chroot, ignoring request: list-timers"
In the sensu_client container deployed by tripleo behaviour is indeed incorrect: [root@controller-0 ~]# podman exec -it sensu_client bash ()[root@controller-0 /]# systemctl list-timers --no-pager --no-legend "tripleo*healthcheck.timer" Running in chroot, ignoring request: list-timers ()[root@controller-0 /]# [root@controller-0 ~]# podman run -it --pid=host --network=host --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/dev/log:/dev/log --volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro --volume=/etc/puppet:/etc/puppet:ro --volume=/var/log/journal:/var/log/journal:ro --volume=/sys/fs/cgroup:/sys/fs/cgroup --volume=/run/dbus/system_bus_socket://run/dbus/system_bus_socket:rw,z --volume=/run:/run --volume=/usr/lib/systemd:/usr/lib/systemd:rw --volume=/var/lib/kolla/config_files/sensu-client.json:/var/lib/kolla/config_files/config.json:ro --volume=/var/lib/config-data/puppet-generated/sensu/:/var/lib/kolla/config_files/src:ro --volume=/var/log/containers/sensu:/var/log/sensu:rw,z ef0425f30d17 bash ()[root@controller-0 /]# systemctl list-timers --no-pager --no-legend "tripleo*healthcheck.timer" Thu 2019-09-05 16:00:49 UTC 16s left Thu 2019-09-05 15:59:47 UTC 45s ago tripleo_gnocchi_api_healthcheck.timer tripleo_gnocchi_api_healthcheck.service Thu 2019-09-05 16:00:49 UTC 16s left Thu 2019-09-05 15:59:47 UTC 45s ago tripleo_heat_api_cron_healthcheck.timer tripleo_heat_api_cron_healthcheck.service Thu 2019-09-05 16:00:52 UTC 19s left Thu 2019-09-05 15:59:47 UTC 45s ago tripleo_heat_api_cfn_healthcheck.timer tripleo_heat_api_cfn_healthcheck.service Thu 2019-09-05 16:00:55 UTC 21s left Thu 2019-09-05 15:59:47 UTC 45s ago tripleo_glance_api_healthcheck.timer tripleo_glance_api_healthcheck.service Thu 2019-09-05 16:01:01 UTC 28s left Thu 2019-09-05 15:59:47 UTC 45s ago tripleo_keystone_healthcheck.timer tripleo_keystone_healthcheck.service Thu 2019-09-05 16:01:04 UTC 31s left Thu 2019-09-05 15:59:47 UTC 45s ago tripleo_nova_consoleauth_healthcheck.timer tripleo_nova_consoleauth_healthcheck.service Thu 2019-09-05 16:01:06 UTC 32s left Thu 2019-09-05 15:59:47 UTC 45s ago tripleo_swift_object_server_healthcheck.timer tripleo_swift_object_server_healthcheck.service Thu 2019-09-05 16:01:06 UTC 33s left Thu 2019-09-05 15:59:47 UTC 45s ago tripleo_memcached_healthcheck.timer tripleo_memcached_healthcheck.service Thu 2019-09-05 16:01:09 UTC 36s left Thu 2019-09-05 15:59:47 UTC 45s ago tripleo_ceilometer_agent_notification_healthcheck.timer tripleo_ceilometer_agent_notification_healthcheck.service Thu 2019-09-05 16:01:12 UTC 39s left Thu 2019-09-05 15:59:47 UTC 45s ago tripleo_aodh_api_healthcheck.timer tripleo_aodh_api_healthcheck.service Thu 2019-09-05 16:01:12 UTC 39s left Thu 2019-09-05 15:59:54 UTC 39s ago tripleo_nova_conductor_healthcheck.timer tripleo_nova_conductor_healthcheck.service Thu 2019-09-05 16:01:14 UTC 41s left Thu 2019-09-05 16:00:08 UTC 25s ago tripleo_nova_api_cron_healthcheck.timer tripleo_nova_api_cron_healthcheck.service Thu 2019-09-05 16:01:15 UTC 41s left Thu 2019-09-05 16:00:03 UTC 29s ago tripleo_gnocchi_metricd_healthcheck.timer tripleo_gnocchi_metricd_healthcheck.service Thu 2019-09-05 16:01:15 UTC 42s left Thu 2019-09-05 15:59:47 UTC 45s ago tripleo_panko_api_healthcheck.timer tripleo_panko_api_healthcheck.service Thu 2019-09-05 16:01:16 UTC 43s left Thu 2019-09-05 16:00:03 UTC 29s ago tripleo_aodh_evaluator_healthcheck.timer tripleo_aodh_evaluator_healthcheck.service Thu 2019-09-05 16:01:18 UTC 45s left Thu 2019-09-05 16:00:03 UTC 29s ago tripleo_heat_engine_healthcheck.timer tripleo_heat_engine_healthcheck.service Thu 2019-09-05 16:01:18 UTC 45s left Thu 2019-09-05 15:59:54 UTC 39s ago tripleo_ovn_controller_healthcheck.timer tripleo_ovn_controller_healthcheck.service Thu 2019-09-05 16:01:19 UTC 46s left Thu 2019-09-05 16:00:03 UTC 29s ago tripleo_sensu_client_healthcheck.timer tripleo_sensu_client_healthcheck.service Thu 2019-09-05 16:01:20 UTC 47s left Thu 2019-09-05 15:59:54 UTC 39s ago tripleo_swift_container_server_healthcheck.timer tripleo_swift_container_server_healthcheck.service Thu 2019-09-05 16:01:21 UTC 47s left Thu 2019-09-05 15:59:47 UTC 45s ago tripleo_neutron_api_healthcheck.timer tripleo_neutron_api_healthcheck.service Thu 2019-09-05 16:01:21 UTC 48s left Thu 2019-09-05 15:59:54 UTC 39s ago tripleo_iscsid_healthcheck.timer tripleo_iscsid_healthcheck.service Thu 2019-09-05 16:01:22 UTC 49s left Thu 2019-09-05 15:59:47 UTC 45s ago tripleo_nova_scheduler_healthcheck.timer tripleo_nova_scheduler_healthcheck.service Thu 2019-09-05 16:01:24 UTC 51s left Thu 2019-09-05 15:59:47 UTC 45s ago tripleo_heat_api_healthcheck.timer tripleo_heat_api_healthcheck.service Thu 2019-09-05 16:01:25 UTC 52s left Thu 2019-09-05 16:00:03 UTC 29s ago tripleo_cinder_api_cron_healthcheck.timer tripleo_cinder_api_cron_healthcheck.service Thu 2019-09-05 16:01:29 UTC 55s left Thu 2019-09-05 15:59:47 UTC 45s ago tripleo_keystone_cron_healthcheck.timer tripleo_keystone_cron_healthcheck.service Thu 2019-09-05 16:01:29 UTC 56s left Thu 2019-09-05 16:00:03 UTC 29s ago tripleo_ceilometer_agent_central_healthcheck.timer tripleo_ceilometer_agent_central_healthcheck.service Thu 2019-09-05 16:01:29 UTC 56s left Thu 2019-09-05 15:59:47 UTC 45s ago tripleo_cinder_scheduler_healthcheck.timer tripleo_cinder_scheduler_healthcheck.service Thu 2019-09-05 16:01:30 UTC 57s left Thu 2019-09-05 16:00:24 UTC 8s ago tripleo_nova_vnc_proxy_healthcheck.timer tripleo_nova_vnc_proxy_healthcheck.service Thu 2019-09-05 16:01:31 UTC 57s left Thu 2019-09-05 15:59:47 UTC 45s ago tripleo_aodh_notifier_healthcheck.timer tripleo_aodh_notifier_healthcheck.service Thu 2019-09-05 16:01:31 UTC 58s left Thu 2019-09-05 16:00:03 UTC 29s ago tripleo_nova_placement_healthcheck.timer tripleo_nova_placement_healthcheck.service Thu 2019-09-05 16:01:33 UTC 1min 0s left Thu 2019-09-05 16:00:24 UTC 8s ago tripleo_logrotate_crond_healthcheck.timer tripleo_logrotate_crond_healthcheck.service Thu 2019-09-05 16:01:35 UTC 1min 2s left Thu 2019-09-05 15:59:54 UTC 39s ago tripleo_cinder_api_healthcheck.timer tripleo_cinder_api_healthcheck.service Thu 2019-09-05 16:01:35 UTC 1min 2s left Thu 2019-09-05 15:59:54 UTC 39s ago tripleo_gnocchi_statsd_healthcheck.timer tripleo_gnocchi_statsd_healthcheck.service Thu 2019-09-05 16:01:44 UTC 1min 11s left Thu 2019-09-05 16:00:08 UTC 25s ago tripleo_nova_metadata_healthcheck.timer tripleo_nova_metadata_healthcheck.service Thu 2019-09-05 16:01:55 UTC 1min 22s left Thu 2019-09-05 16:00:24 UTC 8s ago tripleo_swift_proxy_healthcheck.timer tripleo_swift_proxy_healthcheck.service Thu 2019-09-05 16:02:02 UTC 1min 28s left Thu 2019-09-05 16:00:24 UTC 8s ago tripleo_nova_api_healthcheck.timer tripleo_nova_api_healthcheck.service Thu 2019-09-05 16:02:03 UTC 1min 30s left Thu 2019-09-05 16:00:24 UTC 8s ago tripleo_aodh_listener_healthcheck.timer tripleo_aodh_listener_healthcheck.service Thu 2019-09-05 16:02:08 UTC 1min 35s left Thu 2019-09-05 16:00:24 UTC 8s ago tripleo_swift_account_server_healthcheck.timer tripleo_swift_account_server_healthcheck.service n/a n/a n/a n/a tripleo_collectd_healthcheck.timer tripleo_collectd_healthcheck.service ()[root@controller-0 /]# We will need to investigate on differences. Attached inspect outputs of both containers from Leonids env.
Created attachment 1612040 [details] Containers inspection
Found the cause. The reason is that TripleO deploys container as privileged: [root@controller-0 ~]# podman run --pid=host --network=host --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/dev/log:/dev/log --volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro --volume=/etc/puppet:/etc/puppet:ro --volume=/var/log/journal:/var/log/journal:ro --volume=/sys/fs/cgroup:/sys/fs/cgroup:ro --volume=/run:/run:rw --volume=/usr/lib/systemd/system:/usr/lib/systemd/system:rw --volume=/var/lib/kolla/config_files/sensu-client.json:/var/lib/kolla/config_files/config.json:ro --volume=/var/lib/config-data/puppet-generated/sensu/:/var/lib/kolla/config_files/src:ro --volume=/var/log/containers/sensu:/var/log/sensu:rw,z ef0425f30d17 systemctl list-timers --no-pager --no-legend "tripleo*healthcheck.timer" Thu 2019-09-05 19:04:42 UTC 7s left Thu 2019-09-05 19:03:24 UTC 1min 10s ago tripleo_sensu_client_healthcheck.timer tripleo_sensu_client_healthcheck.service Thu 2019-09-05 19:04:42 UTC 7s left Thu 2019-09-05 19:03:24 UTC 1min 10s ago tripleo_gnocchi_api_healthcheck.timer tripleo_gnocchi_api_healthcheck.service Thu 2019-09-05 19:04:45 UTC 10s left Thu 2019-09-05 19:03:06 UTC 1min 28s ago tripleo_nova_consoleauth_healthcheck.timer tripleo_nova_consoleauth_healthcheck.service Thu 2019-09-05 19:04:45 UTC 10s left Thu 2019-09-05 19:03:06 UTC 1min 28s ago tripleo_nova_placement_healthcheck.timer tripleo_nova_placement_healthcheck.service Thu 2019-09-05 19:04:50 UTC 15s left Thu 2019-09-05 19:03:06 UTC 1min 28s ago tripleo_ceilometer_agent_central_healthcheck.timer tripleo_ceilometer_agent_central_healthcheck.service Thu 2019-09-05 19:04:50 UTC 15s left Thu 2019-09-05 19:03:06 UTC 1min 28s ago tripleo_logrotate_crond_healthcheck.timer tripleo_logrotate_crond_healthcheck.service Thu 2019-09-05 19:04:51 UTC 16s left Thu 2019-09-05 19:03:24 UTC 1min 10s ago tripleo_keystone_cron_healthcheck.timer tripleo_keystone_cron_healthcheck.service Thu 2019-09-05 19:04:54 UTC 18s left Thu 2019-09-05 19:03:24 UTC 1min 10s ago tripleo_gnocchi_metricd_healthcheck.timer tripleo_gnocchi_metricd_healthcheck.service Thu 2019-09-05 19:05:01 UTC 26s left Thu 2019-09-05 19:03:24 UTC 1min 10s ago tripleo_swift_proxy_healthcheck.timer tripleo_swift_proxy_healthcheck.service Thu 2019-09-05 19:05:06 UTC 31s left Thu 2019-09-05 19:03:24 UTC 1min 10s ago tripleo_aodh_evaluator_healthcheck.timer tripleo_aodh_evaluator_healthcheck.service Thu 2019-09-05 19:05:36 UTC 1min 1s left Thu 2019-09-05 19:04:34 UTC 776ms ago tripleo_gnocchi_statsd_healthcheck.timer tripleo_gnocchi_statsd_healthcheck.service Thu 2019-09-05 19:05:37 UTC 1min 2s left Thu 2019-09-05 19:04:34 UTC 779ms ago tripleo_keystone_healthcheck.timer tripleo_keystone_healthcheck.service Thu 2019-09-05 19:05:39 UTC 1min 4s left Thu 2019-09-05 19:04:34 UTC 776ms ago tripleo_panko_api_healthcheck.timer tripleo_panko_api_healthcheck.service Thu 2019-09-05 19:05:42 UTC 1min 7s left Thu 2019-09-05 19:04:34 UTC 776ms ago tripleo_aodh_api_healthcheck.timer tripleo_aodh_api_healthcheck.service Thu 2019-09-05 19:05:46 UTC 1min 11s left Thu 2019-09-05 19:04:34 UTC 778ms ago tripleo_ceilometer_agent_notification_healthcheck.timer tripleo_ceilometer_agent_notification_healthcheck.service Thu 2019-09-05 19:05:50 UTC 1min 15s left Thu 2019-09-05 19:04:34 UTC 777ms ago tripleo_heat_engine_healthcheck.timer tripleo_heat_engine_healthcheck.service Thu 2019-09-05 19:05:51 UTC 1min 16s left Thu 2019-09-05 19:04:34 UTC 779ms ago tripleo_memcached_healthcheck.timer tripleo_memcached_healthcheck.service Thu 2019-09-05 19:05:53 UTC 1min 18s left Thu 2019-09-05 19:04:34 UTC 780ms ago tripleo_nova_metadata_healthcheck.timer tripleo_nova_metadata_healthcheck.service Thu 2019-09-05 19:05:55 UTC 1min 20s left Thu 2019-09-05 19:04:34 UTC 779ms ago tripleo_heat_api_healthcheck.timer tripleo_heat_api_healthcheck.service Thu 2019-09-05 19:05:55 UTC 1min 20s left Thu 2019-09-05 19:04:34 UTC 780ms ago tripleo_heat_api_cfn_healthcheck.timer tripleo_heat_api_cfn_healthcheck.service Thu 2019-09-05 19:06:01 UTC 1min 25s left Thu 2019-09-05 19:04:34 UTC 778ms ago tripleo_cinder_api_cron_healthcheck.timer tripleo_cinder_api_cron_healthcheck.service Thu 2019-09-05 19:06:07 UTC 1min 32s left Thu 2019-09-05 19:04:34 UTC 776ms ago tripleo_heat_api_cron_healthcheck.timer tripleo_heat_api_cron_healthcheck.service Thu 2019-09-05 19:06:09 UTC 1min 33s left Thu 2019-09-05 19:04:34 UTC 780ms ago tripleo_swift_container_server_healthcheck.timer tripleo_swift_container_server_healthcheck.service Thu 2019-09-05 19:06:09 UTC 1min 34s left Thu 2019-09-05 19:04:34 UTC 780ms ago tripleo_nova_scheduler_healthcheck.timer tripleo_nova_scheduler_healthcheck.service Thu 2019-09-05 19:06:09 UTC 1min 34s left Thu 2019-09-05 19:04:34 UTC 778ms ago tripleo_nova_api_healthcheck.timer tripleo_nova_api_healthcheck.service Thu 2019-09-05 19:06:09 UTC 1min 34s left Thu 2019-09-05 19:04:34 UTC 779ms ago tripleo_iscsid_healthcheck.timer tripleo_iscsid_healthcheck.service Thu 2019-09-05 19:06:10 UTC 1min 34s left Thu 2019-09-05 19:04:34 UTC 777ms ago tripleo_cinder_api_healthcheck.timer tripleo_cinder_api_healthcheck.service Thu 2019-09-05 19:06:12 UTC 1min 36s left Thu 2019-09-05 19:04:34 UTC 780ms ago tripleo_aodh_listener_healthcheck.timer tripleo_aodh_listener_healthcheck.service Thu 2019-09-05 19:06:13 UTC 1min 37s left Thu 2019-09-05 19:04:34 UTC 777ms ago tripleo_cinder_scheduler_healthcheck.timer tripleo_cinder_scheduler_healthcheck.service Thu 2019-09-05 19:06:16 UTC 1min 40s left Thu 2019-09-05 19:04:34 UTC 778ms ago tripleo_nova_api_cron_healthcheck.timer tripleo_nova_api_cron_healthcheck.service n/a n/a Thu 2019-09-05 19:04:34 UTC 779ms ago tripleo_aodh_notifier_healthcheck.timer tripleo_aodh_notifier_healthcheck.service n/a n/a n/a n/a tripleo_collectd_healthcheck.timer tripleo_collectd_healthcheck.service n/a n/a Thu 2019-09-05 19:04:34 UTC 780ms ago tripleo_glance_api_healthcheck.timer tripleo_glance_api_healthcheck.service n/a n/a Thu 2019-09-05 19:04:34 UTC 777ms ago tripleo_neutron_api_healthcheck.timer tripleo_neutron_api_healthcheck.service n/a n/a Thu 2019-09-05 19:04:34 UTC 780ms ago tripleo_nova_conductor_healthcheck.timer tripleo_nova_conductor_healthcheck.service n/a n/a Thu 2019-09-05 19:04:34 UTC 777ms ago tripleo_nova_vnc_proxy_healthcheck.timer tripleo_nova_vnc_proxy_healthcheck.service n/a n/a Thu 2019-09-05 19:04:34 UTC 780ms ago tripleo_ovn_controller_healthcheck.timer tripleo_ovn_controller_healthcheck.service n/a n/a Thu 2019-09-05 19:04:34 UTC 778ms ago tripleo_swift_account_server_healthcheck.timer tripleo_swift_account_server_healthcheck.service n/a n/a Thu 2019-09-05 19:04:34 UTC 778ms ago tripleo_swift_object_server_healthcheck.timer tripleo_swift_object_server_healthcheck.service [root@controller-0 ~]# podman run --privileged --pid=host --network=host --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/dev/log:/dev/log --volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro --volume=/etc/puppet:/etc/puppet:ro --volume=/var/log/journal:/var/log/journal:ro --volume=/sys/fs/cgroup:/sys/fs/cgroup:ro --volume=/run:/run:rw --volume=/usr/lib/systemd/system:/usr/lib/systemd/system:rw --volume=/var/lib/kolla/config_files/sensu-client.json:/var/lib/kolla/config_files/config.json:ro --volume=/var/lib/config-data/puppet-generated/sensu/:/var/lib/kolla/config_files/src:ro --volume=/var/log/containers/sensu:/var/log/sensu:rw,z ef0425f30d17 systemctl list-timers --no-pager --no-legend "tripleo*healthcheck.timer" WARNING: The same type, major and minor should not be used for multiple devices. WARNING: The same type, major and minor should not be used for multiple devices. WARNING: The same type, major and minor should not be used for multiple devices. Running in chroot, ignoring request: list-timers [root@controller-0 ~]#
Closing this as this issue was created by not having proper systemd access from sensu-client container. This is definitely fixed and should be in MODIFIED, but since my fix was CVE we removed this and implemented the container-health-check differently.