Bug 1479626
| Summary: | rhosp-director: OSP12 heat and keystone cron jobs are running directly on OC controllers and inside containers. | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Alexander Chuzhoy <sasha> |
| Component: | openstack-tripleo-heat-templates | Assignee: | Dan Prince <dprince> |
| Status: | CLOSED ERRATA | QA Contact: | Artem Hrechanychenko <ahrechan> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 12.0 (Pike) | CC: | dbecker, dprince, m.andre, mburns, mcornea, morazi, ohochman, rhel-osp-director-maint, tvignaud |
| Target Milestone: | beta | Keywords: | Triaged |
| Target Release: | 12.0 (Pike) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-tripleo-heat-templates-7.0.3-0.20171019091937.bbf25a1.el7ost | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-12-13 21:51:30 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
We have a fix posted for this upstream here: https://review.openstack.org/#/c/490496/ It has taken a while to land due to CI issues. Hopefully it will land this week. See also the same issue filed upstream: https://bugs.launchpad.net/tripleo/+bug/1708466 FailedQA
(undercloud) [stack@undercloud-0 ~]$ for i in `openstack server list -f value -c Networks |awk -F'=' '{print $2}'`; do echo "############################################"; ssh -o StrictHostKeyChecking=no heat-admin@$i "hostname; sudo ls /var/spool/cron/"; done
############################################
Warning: Permanently added '192.168.24.20' (ECDSA) to the list of known hosts.
overcloud-controller-2
keystone
############################################
Warning: Permanently added '192.168.24.19' (ECDSA) to the list of known hosts.
overcloud-controller-0
keystone
############################################
Warning: Permanently added '192.168.24.6' (ECDSA) to the list of known hosts.
overcloud-cephstorage-2
############################################
Warning: Permanently added '192.168.24.15' (ECDSA) to the list of known hosts.
overcloud-cephstorage-0
############################################
Warning: Permanently added '192.168.24.9' (ECDSA) to the list of known hosts.
overcloud-controller-1
keystone
############################################
overcloud-compute-0
############################################
Warning: Permanently added '192.168.24.7' (ECDSA) to the list of known hosts.
overcloud-cephstorage-1
############################################
overcloud-compute-1
(undercloud) [stack@undercloud-0 ~]$
(undercloud) [stack@undercloud-0 ~]$ for i in `openstack server list -f value -c Networks |awk -F'=' '{print $2}'`; do echo "############################################"; ssh -o StrictHostKeyChecking=no heat-admin@$i "hostname; sudo docker ps|grep cron"; done
############################################
overcloud-controller-2
e5f642faa726 192.168.24.1:8787/rhosp12/openstack-heat-api-docker:20171026.1 "kolla_start" About an hour ago Up About an hour (healthy) heat_api_cron
ad45153adb6f 192.168.24.1:8787/rhosp12/openstack-cron-docker:20171026.1 "kolla_start" About an hour ago Up About an hour logrotate_crond
5d9b584769dc 192.168.24.1:8787/rhosp12/openstack-nova-api-docker:20171026.1 "kolla_start" About an hour ago Up About an hour (healthy) nova_api_cron
71acbe045a87 192.168.24.1:8787/rhosp12/openstack-keystone-docker:20171026.1 "/bin/bash -c '/usr/l" About an hour ago Up About an hour (healthy) keystone_cron
############################################
overcloud-controller-0
829a91da6a48 192.168.24.1:8787/rhosp12/openstack-heat-api-docker:20171026.1 "kolla_start" About an hour ago Up About an hour (healthy) heat_api_cron
55acc7b95774 192.168.24.1:8787/rhosp12/openstack-cron-docker:20171026.1 "kolla_start" About an hour ago Up About an hour logrotate_crond
12ca3967363e 192.168.24.1:8787/rhosp12/openstack-nova-api-docker:20171026.1 "kolla_start" About an hour ago Up About an hour (healthy) nova_api_cron
3220bb4ccd24 192.168.24.1:8787/rhosp12/openstack-keystone-docker:20171026.1 "/bin/bash -c '/usr/l" About an hour ago Up About an hour (healthy) keystone_cron
############################################
overcloud-cephstorage-2
03e634200b4c 192.168.24.1:8787/rhosp12/openstack-cron-docker:20171026.1 "kolla_start" About an hour ago Up About an hour logrotate_crond
############################################
overcloud-cephstorage-0
477c03495559 192.168.24.1:8787/rhosp12/openstack-cron-docker:20171026.1 "kolla_start" About an hour ago Up About an hour logrotate_crond
############################################
overcloud-controller-1
f12e669017cf 192.168.24.1:8787/rhosp12/openstack-heat-api-docker:20171026.1 "kolla_start" About an hour ago Up About an hour (healthy) heat_api_cron
b0c08a9f69da 192.168.24.1:8787/rhosp12/openstack-cron-docker:20171026.1 "kolla_start" About an hour ago Up About an hour logrotate_crond
e11613aad35b 192.168.24.1:8787/rhosp12/openstack-nova-api-docker:20171026.1 "kolla_start" About an hour ago Up About an hour (healthy) nova_api_cron
ec331f5ce148 192.168.24.1:8787/rhosp12/openstack-keystone-docker:20171026.1 "/bin/bash -c '/usr/l" About an hour ago Up About an hour (healthy) keystone_cron
############################################
overcloud-compute-0
167f3fc05d4d 192.168.24.1:8787/rhosp12/openstack-cron-docker:20171026.1 "kolla_start" About an hour ago Up About an hour logrotate_crond
############################################
overcloud-cephstorage-1
de1c1ffe76e9 192.168.24.1:8787/rhosp12/openstack-cron-docker:20171026.1 "kolla_start" About an hour ago Up About an hour logrotate_crond
############################################
overcloud-compute-1
6105fe4d70f7 192.168.24.1:8787/rhosp12/openstack-cron-docker:20171026.1 "kolla_start" About an hour ago Up About an hour logrotate_crond
(undercloud) [stack@undercloud-0 ~]$ for i in `openstack server list -f value -c Name -c Networks |awk -F'=' '/controller/ {print $2}'`; do echo "############################################"; ssh -o StrictHostKeyChecking=no heat-admin@$i "hostname; for j in \`sudo docker ps|awk '/cron/ {print \$1}'\`; do echo \$j; sudo docker exec -u root \$j bash -c 'ls /var/spool/cron'; done"; done
############################################
overcloud-controller-2
e5f642faa726
heat
ad45153adb6f
root
5d9b584769dc
nova
71acbe045a87
keystone
############################################
overcloud-controller-0
829a91da6a48
heat
55acc7b95774
root
12ca3967363e
nova
3220bb4ccd24
keystone
############################################
overcloud-controller-1
f12e669017cf
heat
b0c08a9f69da
root
e11613aad35b
nova
ec331f5ce148
keystone
vi /var/spool/cron/keystone # HEADER: This file was autogenerated at 2017-10-31 18:01:52 +0000 by puppet. # HEADER: While it can still be managed manually, it is definitely not recommended. # HEADER: Note particularly that the comments starting with 'Puppet Name' should # HEADER: not be deleted, as doing so could cause duplicate cron jobs. # Puppet Name: cinder-manage db purge PATH=/bin:/usr/bin:/usr/sbin SHELL=/bin/sh 1 0 * * * cinder-manage db purge 0 >>/var/log/cinder/cinder-rowsflush.log 2>&1 This fails downstream still because there were two related upstream patches to fix the issue. In the first patch we missed added the Ansible tags so that the new task runs at step 2. See: https://review.openstack.org/#/c/497936/ I added an external tracker for this and will double check to make sure it gets backported. Submitted the patch upstream for backporting to Pike: https://review.openstack.org/517082 https://review.openstack.org/517082 was merged in stable/pike. VERIFIED
openstack-tripleo-heat-templates-7.0.3-3.el7ost.noarch
undercloud) [stack@undercloud-0 ~]$ for i in `openstack server list -f value -c Networks |awk -F'=' '{print $2}'`; do echo "############################################"; ssh -o StrictHostKeyChecking=no heat-admin@$i "hostname; sudo ls /var/spool/cron/"; done
############################################
overcloud-controller-2
cinder
############################################
overcloud-controller-0
cinder
############################################
overcloud-controller-1
cinder
############################################
overcloud-compute-0
############################################
Warning: Permanently added '192.168.24.11' (ECDSA) to the list of known hosts.
overcloud-compute-1
(undercloud) [stack@undercloud-0 ~]$ for i in `openstack server list -f value -c Networks |awk -F'=' '{print $2}'`; do echo "############################################"; ssh -o StrictHostKeyChecking=no heat-admin@$i "hostname; sudo docker ps|grep cron"; done
############################################
overcloud-controller-2
d947707d1a33 192.168.24.1:8787/rhosp12/openstack-heat-api-docker:2017-11-14.4 "kolla_start" 12 hours ago Up 12 hours (healthy) heat_api_cron
9dcbafd658b6 192.168.24.1:8787/rhosp12/openstack-cron-docker:2017-11-14.4 "kolla_start" 12 hours ago Up 12 hours logrotate_crond
ed8146918768 192.168.24.1:8787/rhosp12/openstack-nova-api-docker:2017-11-14.4 "kolla_start" 13 hours ago Up 13 hours (healthy) nova_api_cron
a46a64801e4f 192.168.24.1:8787/rhosp12/openstack-keystone-docker:2017-11-14.4 "/bin/bash -c '/usr/l" 13 hours ago Up 13 hours (healthy) keystone_cron
############################################
overcloud-controller-0
b8ae251e37f3 192.168.24.1:8787/rhosp12/openstack-heat-api-docker:2017-11-14.4 "kolla_start" 12 hours ago Up 12 hours (healthy) heat_api_cron
8449f9fde865 192.168.24.1:8787/rhosp12/openstack-cron-docker:2017-11-14.4 "kolla_start" 12 hours ago Up 12 hours logrotate_crond
5782fc9098a8 192.168.24.1:8787/rhosp12/openstack-nova-api-docker:2017-11-14.4 "kolla_start" 12 hours ago Up 12 hours (healthy) nova_api_cron
b9d41546e865 192.168.24.1:8787/rhosp12/openstack-keystone-docker:2017-11-14.4 "/bin/bash -c '/usr/l" 13 hours ago Up 13 hours (healthy) keystone_cron
############################################
overcloud-controller-1
19f0dde2ac00 192.168.24.1:8787/rhosp12/openstack-heat-api-docker:2017-11-14.4 "kolla_start" 12 hours ago Up 12 hours (healthy) heat_api_cron
8ae65be2631c 192.168.24.1:8787/rhosp12/openstack-cron-docker:2017-11-14.4 "kolla_start" 12 hours ago Up 12 hours logrotate_crond
1371f7b0dffd 192.168.24.1:8787/rhosp12/openstack-nova-api-docker:2017-11-14.4 "kolla_start" 13 hours ago Up 13 hours (healthy) nova_api_cron
edf1489ac5e8 192.168.24.1:8787/rhosp12/openstack-keystone-docker:2017-11-14.4 "/bin/bash -c '/usr/l" 13 hours ago Up 13 hours (healthy) keystone_cron
############################################
overcloud-compute-0
aa4c78b7ee4d 192.168.24.1:8787/rhosp12/openstack-cron-docker:2017-11-14.4 "kolla_start" 13 hours ago Up 17 minutes logrotate_crond
############################################
overcloud-compute-1
bdefb26d499a 192.168.24.1:8787/rhosp12/openstack-cron-docker:2017-11-14.4 "kolla_start" 13 hours ago Up 13 hours logrotate_crond
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3462 |
rhosp-director: OSP12 heat and keystone cron jobs are running directly on OC controllers and inside containers. Environment: Steps to reproduce: 1. Deploy OC. 2. Check the cron spool dir on all nodes. Result: Show the content of /var/spool/cron dir on every node: (undercloud) [stack@undercloud-0 ~]$ for i in `openstack server list -f value -c Networks |awk -F'=' '{print $2}'`; do echo "############################################"; ssh -o StrictHostKeyChecking=no heat-admin@$i "hostname; sudo ls /var/spool/cron/"; done ############################################ controller-2 heat keystone ############################################ ceph-2 ############################################ controller-0 heat keystone ############################################ controller-1 heat keystone ############################################ ceph-0 ############################################ compute-0 ############################################ ceph-1 ############################################ compute-1 Show containers with name including the string 'cron' on all nodes: (undercloud) [stack@undercloud-0 ~]$ for i in `openstack server list -f value -c Networks |awk -F'=' '{print $2}'`; do echo "############################################"; ssh -o StrictHostKeyChecking=no heat-admin@$i "hostname; sudo docker ps|grep cron"; done ############################################ controller-2 bcdc9f822540 192.168.24.1:8787/rhosp12/openstack-heat-api-docker:2017-08-07.4 "kolla_start" 3 hours ago Up 3 hours heat_api_cron 62f725d4ee99 192.168.24.1:8787/rhosp12/openstack-nova-api-docker:2017-08-07.4 "kolla_start" 3 hours ago Up 3 hours nova_api_cron 514a2bcf9462 192.168.24.1:8787/rhosp12/openstack-keystone-docker:2017-08-07.4 "/bin/bash -c '/usr/l" 3 hours ago Up 3 hours keystone_cron ############################################ ceph-2 ############################################ controller-0 cd34bde93503 192.168.24.1:8787/rhosp12/openstack-heat-api-docker:2017-08-07.4 "kolla_start" 3 hours ago Up 3 hours heat_api_cron 8eef2736ecf2 192.168.24.1:8787/rhosp12/openstack-nova-api-docker:2017-08-07.4 "kolla_start" 3 hours ago Up 3 hours nova_api_cron 8c7c3992528b 192.168.24.1:8787/rhosp12/openstack-keystone-docker:2017-08-07.4 "/bin/bash -c '/usr/l" 3 hours ago Up 3 hours keystone_cron ############################################ controller-1 027b90dcde8e 192.168.24.1:8787/rhosp12/openstack-heat-api-docker:2017-08-07.4 "kolla_start" 3 hours ago Up 3 hours heat_api_cron a6980f0dc902 192.168.24.1:8787/rhosp12/openstack-nova-api-docker:2017-08-07.4 "kolla_start" 3 hours ago Up 3 hours nova_api_cron 28c2ed628278 192.168.24.1:8787/rhosp12/openstack-keystone-docker:2017-08-07.4 "/bin/bash -c '/usr/l" 3 hours ago Up 3 hours keystone_cron ############################################ ceph-0 ############################################ compute-0 ############################################ ceph-1 ############################################ compute-1 Show the hostname, ID of container (with name including 'cron' string) and content of its /var/spool/cron dir: (undercloud) [stack@undercloud-0 ~]$ for i in `openstack server list -f value -c Name -c Networks |awk -F'=' '/controller/ {print $2}'`; do echo "############################################"; ssh -o StrictHostKeyChecking=no heat-admin@$i "hostname; for j in \`sudo docker ps|awk '/cron/ {print \$1}'\`; do echo \$j; sudo docker exec -u root \$j bash -c 'ls /var/spool/cron'; done"; done ############################################ controller-2 bcdc9f822540 heat 62f725d4ee99 nova 514a2bcf9462 keystone ############################################ controller-0 cd34bde93503 heat 8eef2736ecf2 nova 8c7c3992528b keystone ############################################ controller-1 027b90dcde8e heat a6980f0dc902 nova 28c2ed628278 keystone (undercloud) [stack@undercloud-0 ~]$