rhosp-director: OSP12 heat and keystone cron jobs are running directly on OC controllers and inside containers. Environment: Steps to reproduce: 1. Deploy OC. 2. Check the cron spool dir on all nodes. Result: Show the content of /var/spool/cron dir on every node: (undercloud) [stack@undercloud-0 ~]$ for i in `openstack server list -f value -c Networks |awk -F'=' '{print $2}'`; do echo "############################################"; ssh -o StrictHostKeyChecking=no heat-admin@$i "hostname; sudo ls /var/spool/cron/"; done ############################################ controller-2 heat keystone ############################################ ceph-2 ############################################ controller-0 heat keystone ############################################ controller-1 heat keystone ############################################ ceph-0 ############################################ compute-0 ############################################ ceph-1 ############################################ compute-1 Show containers with name including the string 'cron' on all nodes: (undercloud) [stack@undercloud-0 ~]$ for i in `openstack server list -f value -c Networks |awk -F'=' '{print $2}'`; do echo "############################################"; ssh -o StrictHostKeyChecking=no heat-admin@$i "hostname; sudo docker ps|grep cron"; done ############################################ controller-2 bcdc9f822540 192.168.24.1:8787/rhosp12/openstack-heat-api-docker:2017-08-07.4 "kolla_start" 3 hours ago Up 3 hours heat_api_cron 62f725d4ee99 192.168.24.1:8787/rhosp12/openstack-nova-api-docker:2017-08-07.4 "kolla_start" 3 hours ago Up 3 hours nova_api_cron 514a2bcf9462 192.168.24.1:8787/rhosp12/openstack-keystone-docker:2017-08-07.4 "/bin/bash -c '/usr/l" 3 hours ago Up 3 hours keystone_cron ############################################ ceph-2 ############################################ controller-0 cd34bde93503 192.168.24.1:8787/rhosp12/openstack-heat-api-docker:2017-08-07.4 "kolla_start" 3 hours ago Up 3 hours heat_api_cron 8eef2736ecf2 192.168.24.1:8787/rhosp12/openstack-nova-api-docker:2017-08-07.4 "kolla_start" 3 hours ago Up 3 hours nova_api_cron 8c7c3992528b 192.168.24.1:8787/rhosp12/openstack-keystone-docker:2017-08-07.4 "/bin/bash -c '/usr/l" 3 hours ago Up 3 hours keystone_cron ############################################ controller-1 027b90dcde8e 192.168.24.1:8787/rhosp12/openstack-heat-api-docker:2017-08-07.4 "kolla_start" 3 hours ago Up 3 hours heat_api_cron a6980f0dc902 192.168.24.1:8787/rhosp12/openstack-nova-api-docker:2017-08-07.4 "kolla_start" 3 hours ago Up 3 hours nova_api_cron 28c2ed628278 192.168.24.1:8787/rhosp12/openstack-keystone-docker:2017-08-07.4 "/bin/bash -c '/usr/l" 3 hours ago Up 3 hours keystone_cron ############################################ ceph-0 ############################################ compute-0 ############################################ ceph-1 ############################################ compute-1 Show the hostname, ID of container (with name including 'cron' string) and content of its /var/spool/cron dir: (undercloud) [stack@undercloud-0 ~]$ for i in `openstack server list -f value -c Name -c Networks |awk -F'=' '/controller/ {print $2}'`; do echo "############################################"; ssh -o StrictHostKeyChecking=no heat-admin@$i "hostname; for j in \`sudo docker ps|awk '/cron/ {print \$1}'\`; do echo \$j; sudo docker exec -u root \$j bash -c 'ls /var/spool/cron'; done"; done ############################################ controller-2 bcdc9f822540 heat 62f725d4ee99 nova 514a2bcf9462 keystone ############################################ controller-0 cd34bde93503 heat 8eef2736ecf2 nova 8c7c3992528b keystone ############################################ controller-1 027b90dcde8e heat a6980f0dc902 nova 28c2ed628278 keystone (undercloud) [stack@undercloud-0 ~]$
We have a fix posted for this upstream here: https://review.openstack.org/#/c/490496/ It has taken a while to land due to CI issues. Hopefully it will land this week. See also the same issue filed upstream: https://bugs.launchpad.net/tripleo/+bug/1708466
FailedQA (undercloud) [stack@undercloud-0 ~]$ for i in `openstack server list -f value -c Networks |awk -F'=' '{print $2}'`; do echo "############################################"; ssh -o StrictHostKeyChecking=no heat-admin@$i "hostname; sudo ls /var/spool/cron/"; done ############################################ Warning: Permanently added '192.168.24.20' (ECDSA) to the list of known hosts. overcloud-controller-2 keystone ############################################ Warning: Permanently added '192.168.24.19' (ECDSA) to the list of known hosts. overcloud-controller-0 keystone ############################################ Warning: Permanently added '192.168.24.6' (ECDSA) to the list of known hosts. overcloud-cephstorage-2 ############################################ Warning: Permanently added '192.168.24.15' (ECDSA) to the list of known hosts. overcloud-cephstorage-0 ############################################ Warning: Permanently added '192.168.24.9' (ECDSA) to the list of known hosts. overcloud-controller-1 keystone ############################################ overcloud-compute-0 ############################################ Warning: Permanently added '192.168.24.7' (ECDSA) to the list of known hosts. overcloud-cephstorage-1 ############################################ overcloud-compute-1 (undercloud) [stack@undercloud-0 ~]$ (undercloud) [stack@undercloud-0 ~]$ for i in `openstack server list -f value -c Networks |awk -F'=' '{print $2}'`; do echo "############################################"; ssh -o StrictHostKeyChecking=no heat-admin@$i "hostname; sudo docker ps|grep cron"; done ############################################ overcloud-controller-2 e5f642faa726 192.168.24.1:8787/rhosp12/openstack-heat-api-docker:20171026.1 "kolla_start" About an hour ago Up About an hour (healthy) heat_api_cron ad45153adb6f 192.168.24.1:8787/rhosp12/openstack-cron-docker:20171026.1 "kolla_start" About an hour ago Up About an hour logrotate_crond 5d9b584769dc 192.168.24.1:8787/rhosp12/openstack-nova-api-docker:20171026.1 "kolla_start" About an hour ago Up About an hour (healthy) nova_api_cron 71acbe045a87 192.168.24.1:8787/rhosp12/openstack-keystone-docker:20171026.1 "/bin/bash -c '/usr/l" About an hour ago Up About an hour (healthy) keystone_cron ############################################ overcloud-controller-0 829a91da6a48 192.168.24.1:8787/rhosp12/openstack-heat-api-docker:20171026.1 "kolla_start" About an hour ago Up About an hour (healthy) heat_api_cron 55acc7b95774 192.168.24.1:8787/rhosp12/openstack-cron-docker:20171026.1 "kolla_start" About an hour ago Up About an hour logrotate_crond 12ca3967363e 192.168.24.1:8787/rhosp12/openstack-nova-api-docker:20171026.1 "kolla_start" About an hour ago Up About an hour (healthy) nova_api_cron 3220bb4ccd24 192.168.24.1:8787/rhosp12/openstack-keystone-docker:20171026.1 "/bin/bash -c '/usr/l" About an hour ago Up About an hour (healthy) keystone_cron ############################################ overcloud-cephstorage-2 03e634200b4c 192.168.24.1:8787/rhosp12/openstack-cron-docker:20171026.1 "kolla_start" About an hour ago Up About an hour logrotate_crond ############################################ overcloud-cephstorage-0 477c03495559 192.168.24.1:8787/rhosp12/openstack-cron-docker:20171026.1 "kolla_start" About an hour ago Up About an hour logrotate_crond ############################################ overcloud-controller-1 f12e669017cf 192.168.24.1:8787/rhosp12/openstack-heat-api-docker:20171026.1 "kolla_start" About an hour ago Up About an hour (healthy) heat_api_cron b0c08a9f69da 192.168.24.1:8787/rhosp12/openstack-cron-docker:20171026.1 "kolla_start" About an hour ago Up About an hour logrotate_crond e11613aad35b 192.168.24.1:8787/rhosp12/openstack-nova-api-docker:20171026.1 "kolla_start" About an hour ago Up About an hour (healthy) nova_api_cron ec331f5ce148 192.168.24.1:8787/rhosp12/openstack-keystone-docker:20171026.1 "/bin/bash -c '/usr/l" About an hour ago Up About an hour (healthy) keystone_cron ############################################ overcloud-compute-0 167f3fc05d4d 192.168.24.1:8787/rhosp12/openstack-cron-docker:20171026.1 "kolla_start" About an hour ago Up About an hour logrotate_crond ############################################ overcloud-cephstorage-1 de1c1ffe76e9 192.168.24.1:8787/rhosp12/openstack-cron-docker:20171026.1 "kolla_start" About an hour ago Up About an hour logrotate_crond ############################################ overcloud-compute-1 6105fe4d70f7 192.168.24.1:8787/rhosp12/openstack-cron-docker:20171026.1 "kolla_start" About an hour ago Up About an hour logrotate_crond (undercloud) [stack@undercloud-0 ~]$ for i in `openstack server list -f value -c Name -c Networks |awk -F'=' '/controller/ {print $2}'`; do echo "############################################"; ssh -o StrictHostKeyChecking=no heat-admin@$i "hostname; for j in \`sudo docker ps|awk '/cron/ {print \$1}'\`; do echo \$j; sudo docker exec -u root \$j bash -c 'ls /var/spool/cron'; done"; done ############################################ overcloud-controller-2 e5f642faa726 heat ad45153adb6f root 5d9b584769dc nova 71acbe045a87 keystone ############################################ overcloud-controller-0 829a91da6a48 heat 55acc7b95774 root 12ca3967363e nova 3220bb4ccd24 keystone ############################################ overcloud-controller-1 f12e669017cf heat b0c08a9f69da root e11613aad35b nova ec331f5ce148 keystone
vi /var/spool/cron/keystone # HEADER: This file was autogenerated at 2017-10-31 18:01:52 +0000 by puppet. # HEADER: While it can still be managed manually, it is definitely not recommended. # HEADER: Note particularly that the comments starting with 'Puppet Name' should # HEADER: not be deleted, as doing so could cause duplicate cron jobs. # Puppet Name: cinder-manage db purge PATH=/bin:/usr/bin:/usr/sbin SHELL=/bin/sh 1 0 * * * cinder-manage db purge 0 >>/var/log/cinder/cinder-rowsflush.log 2>&1
This fails downstream still because there were two related upstream patches to fix the issue. In the first patch we missed added the Ansible tags so that the new task runs at step 2. See: https://review.openstack.org/#/c/497936/ I added an external tracker for this and will double check to make sure it gets backported.
Submitted the patch upstream for backporting to Pike: https://review.openstack.org/517082
https://review.openstack.org/517082 was merged in stable/pike.
VERIFIED openstack-tripleo-heat-templates-7.0.3-3.el7ost.noarch undercloud) [stack@undercloud-0 ~]$ for i in `openstack server list -f value -c Networks |awk -F'=' '{print $2}'`; do echo "############################################"; ssh -o StrictHostKeyChecking=no heat-admin@$i "hostname; sudo ls /var/spool/cron/"; done ############################################ overcloud-controller-2 cinder ############################################ overcloud-controller-0 cinder ############################################ overcloud-controller-1 cinder ############################################ overcloud-compute-0 ############################################ Warning: Permanently added '192.168.24.11' (ECDSA) to the list of known hosts. overcloud-compute-1 (undercloud) [stack@undercloud-0 ~]$ for i in `openstack server list -f value -c Networks |awk -F'=' '{print $2}'`; do echo "############################################"; ssh -o StrictHostKeyChecking=no heat-admin@$i "hostname; sudo docker ps|grep cron"; done ############################################ overcloud-controller-2 d947707d1a33 192.168.24.1:8787/rhosp12/openstack-heat-api-docker:2017-11-14.4 "kolla_start" 12 hours ago Up 12 hours (healthy) heat_api_cron 9dcbafd658b6 192.168.24.1:8787/rhosp12/openstack-cron-docker:2017-11-14.4 "kolla_start" 12 hours ago Up 12 hours logrotate_crond ed8146918768 192.168.24.1:8787/rhosp12/openstack-nova-api-docker:2017-11-14.4 "kolla_start" 13 hours ago Up 13 hours (healthy) nova_api_cron a46a64801e4f 192.168.24.1:8787/rhosp12/openstack-keystone-docker:2017-11-14.4 "/bin/bash -c '/usr/l" 13 hours ago Up 13 hours (healthy) keystone_cron ############################################ overcloud-controller-0 b8ae251e37f3 192.168.24.1:8787/rhosp12/openstack-heat-api-docker:2017-11-14.4 "kolla_start" 12 hours ago Up 12 hours (healthy) heat_api_cron 8449f9fde865 192.168.24.1:8787/rhosp12/openstack-cron-docker:2017-11-14.4 "kolla_start" 12 hours ago Up 12 hours logrotate_crond 5782fc9098a8 192.168.24.1:8787/rhosp12/openstack-nova-api-docker:2017-11-14.4 "kolla_start" 12 hours ago Up 12 hours (healthy) nova_api_cron b9d41546e865 192.168.24.1:8787/rhosp12/openstack-keystone-docker:2017-11-14.4 "/bin/bash -c '/usr/l" 13 hours ago Up 13 hours (healthy) keystone_cron ############################################ overcloud-controller-1 19f0dde2ac00 192.168.24.1:8787/rhosp12/openstack-heat-api-docker:2017-11-14.4 "kolla_start" 12 hours ago Up 12 hours (healthy) heat_api_cron 8ae65be2631c 192.168.24.1:8787/rhosp12/openstack-cron-docker:2017-11-14.4 "kolla_start" 12 hours ago Up 12 hours logrotate_crond 1371f7b0dffd 192.168.24.1:8787/rhosp12/openstack-nova-api-docker:2017-11-14.4 "kolla_start" 13 hours ago Up 13 hours (healthy) nova_api_cron edf1489ac5e8 192.168.24.1:8787/rhosp12/openstack-keystone-docker:2017-11-14.4 "/bin/bash -c '/usr/l" 13 hours ago Up 13 hours (healthy) keystone_cron ############################################ overcloud-compute-0 aa4c78b7ee4d 192.168.24.1:8787/rhosp12/openstack-cron-docker:2017-11-14.4 "kolla_start" 13 hours ago Up 17 minutes logrotate_crond ############################################ overcloud-compute-1 bdefb26d499a 192.168.24.1:8787/rhosp12/openstack-cron-docker:2017-11-14.4 "kolla_start" 13 hours ago Up 13 hours logrotate_crond
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3462