Bug 1479626 - rhosp-director: OSP12 heat and keystone cron jobs are running directly on OC controllers and inside containers.
rhosp-director: OSP12 heat and keystone cron jobs are running directly on OC ...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates (Show other bugs)
12.0 (Pike)
Unspecified Unspecified
high Severity high
: beta
: 12.0 (Pike)
Assigned To: Dan Prince
Artem Hrechanychenko
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-08-08 22:28 EDT by Alexander Chuzhoy
Modified: 2018-02-05 14:12 EST (History)
9 users (show)

See Also:
Fixed In Version: openstack-tripleo-heat-templates-7.0.3-0.20171019091937.bbf25a1.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-12-13 16:51:30 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
OpenStack gerrit 490496 None None None 2017-10-20 08:23 EDT
OpenStack gerrit 497936 None None None 2017-11-01 14:46 EDT
OpenStack gerrit 517082 None None None 2017-11-01 15:05 EDT

  None (edit)
Description Alexander Chuzhoy 2017-08-08 22:28:07 EDT
rhosp-director: OSP12 heat and keystone cron jobs are running directly on OC controllers and inside containers.


Environment:


Steps to reproduce:
1. Deploy OC.
2. Check the cron spool dir on all nodes.


Result:
Show the content of /var/spool/cron dir on every node:

(undercloud) [stack@undercloud-0 ~]$ for i in `openstack server list -f value -c Networks |awk -F'=' '{print $2}'`; do echo "############################################"; ssh -o StrictHostKeyChecking=no heat-admin@$i "hostname; sudo ls /var/spool/cron/"; done
############################################
controller-2
heat
keystone
############################################
ceph-2
############################################
controller-0
heat
keystone
############################################
controller-1
heat
keystone
############################################
ceph-0
############################################
compute-0
############################################
ceph-1
############################################
compute-1




Show containers with name including the string 'cron' on all nodes:

(undercloud) [stack@undercloud-0 ~]$ for i in `openstack server list -f value -c Networks |awk -F'=' '{print $2}'`; do echo "############################################"; ssh -o StrictHostKeyChecking=no heat-admin@$i "hostname; sudo docker ps|grep cron"; done
############################################
controller-2
bcdc9f822540        192.168.24.1:8787/rhosp12/openstack-heat-api-docker:2017-08-07.4                  "kolla_start"            3 hours ago         Up 3 hours                              heat_api_cron
62f725d4ee99        192.168.24.1:8787/rhosp12/openstack-nova-api-docker:2017-08-07.4                  "kolla_start"            3 hours ago         Up 3 hours                              nova_api_cron
514a2bcf9462        192.168.24.1:8787/rhosp12/openstack-keystone-docker:2017-08-07.4                  "/bin/bash -c '/usr/l"   3 hours ago         Up 3 hours                              keystone_cron
############################################
ceph-2
############################################
controller-0
cd34bde93503        192.168.24.1:8787/rhosp12/openstack-heat-api-docker:2017-08-07.4                  "kolla_start"            3 hours ago         Up 3 hours                              heat_api_cron
8eef2736ecf2        192.168.24.1:8787/rhosp12/openstack-nova-api-docker:2017-08-07.4                  "kolla_start"            3 hours ago         Up 3 hours                              nova_api_cron
8c7c3992528b        192.168.24.1:8787/rhosp12/openstack-keystone-docker:2017-08-07.4                  "/bin/bash -c '/usr/l"   3 hours ago         Up 3 hours                              keystone_cron
############################################
controller-1
027b90dcde8e        192.168.24.1:8787/rhosp12/openstack-heat-api-docker:2017-08-07.4                  "kolla_start"            3 hours ago         Up 3 hours                              heat_api_cron
a6980f0dc902        192.168.24.1:8787/rhosp12/openstack-nova-api-docker:2017-08-07.4                  "kolla_start"            3 hours ago         Up 3 hours                              nova_api_cron
28c2ed628278        192.168.24.1:8787/rhosp12/openstack-keystone-docker:2017-08-07.4                  "/bin/bash -c '/usr/l"   3 hours ago         Up 3 hours                              keystone_cron
############################################
ceph-0
############################################
compute-0
############################################
ceph-1
############################################
compute-1



Show the hostname, ID of container (with name including 'cron' string) and content of its /var/spool/cron dir:

(undercloud) [stack@undercloud-0 ~]$ for i in `openstack server list -f value -c Name -c Networks |awk -F'=' '/controller/ {print $2}'`; do echo "############################################"; ssh -o StrictHostKeyChecking=no heat-admin@$i "hostname; for j in \`sudo docker ps|awk '/cron/ {print \$1}'\`; do echo \$j; sudo docker exec -u root \$j bash -c 'ls /var/spool/cron'; done"; done
############################################
controller-2
bcdc9f822540
heat
62f725d4ee99
nova
514a2bcf9462
keystone
############################################
controller-0
cd34bde93503
heat
8eef2736ecf2
nova
8c7c3992528b
keystone
############################################
controller-1
027b90dcde8e
heat
a6980f0dc902
nova
28c2ed628278
keystone
(undercloud) [stack@undercloud-0 ~]$
Comment 1 Dan Prince 2017-08-16 09:09:05 EDT
We have a fix posted for this upstream here:

https://review.openstack.org/#/c/490496/

It has taken a while to land due to CI issues. Hopefully it will land this week.

See also the same issue filed upstream: https://bugs.launchpad.net/tripleo/+bug/1708466
Comment 4 Artem Hrechanychenko 2017-10-31 11:13:45 EDT
FailedQA

(undercloud) [stack@undercloud-0 ~]$ for i in `openstack server list -f value -c Networks |awk -F'=' '{print $2}'`; do echo "############################################"; ssh -o StrictHostKeyChecking=no heat-admin@$i "hostname; sudo ls /var/spool/cron/"; done
############################################
Warning: Permanently added '192.168.24.20' (ECDSA) to the list of known hosts.
overcloud-controller-2
keystone
############################################
Warning: Permanently added '192.168.24.19' (ECDSA) to the list of known hosts.
overcloud-controller-0
keystone
############################################
Warning: Permanently added '192.168.24.6' (ECDSA) to the list of known hosts.
overcloud-cephstorage-2
############################################
Warning: Permanently added '192.168.24.15' (ECDSA) to the list of known hosts.
overcloud-cephstorage-0
############################################
Warning: Permanently added '192.168.24.9' (ECDSA) to the list of known hosts.
overcloud-controller-1
keystone
############################################
overcloud-compute-0
############################################
Warning: Permanently added '192.168.24.7' (ECDSA) to the list of known hosts.
overcloud-cephstorage-1
############################################
overcloud-compute-1
(undercloud) [stack@undercloud-0 ~]$ 


(undercloud) [stack@undercloud-0 ~]$ for i in `openstack server list -f value -c Networks |awk -F'=' '{print $2}'`; do echo "############################################"; ssh -o StrictHostKeyChecking=no heat-admin@$i "hostname; sudo docker ps|grep cron"; done
############################################
overcloud-controller-2
e5f642faa726        192.168.24.1:8787/rhosp12/openstack-heat-api-docker:20171026.1                  "kolla_start"            About an hour ago   Up About an hour (healthy)                         heat_api_cron
ad45153adb6f        192.168.24.1:8787/rhosp12/openstack-cron-docker:20171026.1                      "kolla_start"            About an hour ago   Up About an hour                                   logrotate_crond
5d9b584769dc        192.168.24.1:8787/rhosp12/openstack-nova-api-docker:20171026.1                  "kolla_start"            About an hour ago   Up About an hour (healthy)                         nova_api_cron
71acbe045a87        192.168.24.1:8787/rhosp12/openstack-keystone-docker:20171026.1                  "/bin/bash -c '/usr/l"   About an hour ago   Up About an hour (healthy)                         keystone_cron
############################################
overcloud-controller-0
829a91da6a48        192.168.24.1:8787/rhosp12/openstack-heat-api-docker:20171026.1                  "kolla_start"            About an hour ago   Up About an hour (healthy)                         heat_api_cron
55acc7b95774        192.168.24.1:8787/rhosp12/openstack-cron-docker:20171026.1                      "kolla_start"            About an hour ago   Up About an hour                                   logrotate_crond
12ca3967363e        192.168.24.1:8787/rhosp12/openstack-nova-api-docker:20171026.1                  "kolla_start"            About an hour ago   Up About an hour (healthy)                         nova_api_cron
3220bb4ccd24        192.168.24.1:8787/rhosp12/openstack-keystone-docker:20171026.1                  "/bin/bash -c '/usr/l"   About an hour ago   Up About an hour (healthy)                         keystone_cron
############################################
overcloud-cephstorage-2
03e634200b4c        192.168.24.1:8787/rhosp12/openstack-cron-docker:20171026.1   "kolla_start"       About an hour ago   Up About an hour                        logrotate_crond
############################################
overcloud-cephstorage-0
477c03495559        192.168.24.1:8787/rhosp12/openstack-cron-docker:20171026.1   "kolla_start"       About an hour ago   Up About an hour                        logrotate_crond
############################################
overcloud-controller-1
f12e669017cf        192.168.24.1:8787/rhosp12/openstack-heat-api-docker:20171026.1                  "kolla_start"            About an hour ago   Up About an hour (healthy)                         heat_api_cron
b0c08a9f69da        192.168.24.1:8787/rhosp12/openstack-cron-docker:20171026.1                      "kolla_start"            About an hour ago   Up About an hour                                   logrotate_crond
e11613aad35b        192.168.24.1:8787/rhosp12/openstack-nova-api-docker:20171026.1                  "kolla_start"            About an hour ago   Up About an hour (healthy)                         nova_api_cron
ec331f5ce148        192.168.24.1:8787/rhosp12/openstack-keystone-docker:20171026.1                  "/bin/bash -c '/usr/l"   About an hour ago   Up About an hour (healthy)                         keystone_cron
############################################
overcloud-compute-0
167f3fc05d4d        192.168.24.1:8787/rhosp12/openstack-cron-docker:20171026.1                 "kolla_start"       About an hour ago   Up About an hour                                   logrotate_crond
############################################
overcloud-cephstorage-1
de1c1ffe76e9        192.168.24.1:8787/rhosp12/openstack-cron-docker:20171026.1   "kolla_start"       About an hour ago   Up About an hour                        logrotate_crond
############################################
overcloud-compute-1
6105fe4d70f7        192.168.24.1:8787/rhosp12/openstack-cron-docker:20171026.1                 "kolla_start"       About an hour ago   Up About an hour                                   logrotate_crond


(undercloud) [stack@undercloud-0 ~]$ for i in `openstack server list -f value -c Name -c Networks |awk -F'=' '/controller/ {print $2}'`; do echo "############################################"; ssh -o StrictHostKeyChecking=no heat-admin@$i "hostname; for j in \`sudo docker ps|awk '/cron/ {print \$1}'\`; do echo \$j; sudo docker exec -u root \$j bash -c 'ls /var/spool/cron'; done"; done
############################################
overcloud-controller-2
e5f642faa726
heat
ad45153adb6f
root
5d9b584769dc
nova
71acbe045a87
keystone
############################################
overcloud-controller-0
829a91da6a48
heat
55acc7b95774
root
12ca3967363e
nova
3220bb4ccd24
keystone
############################################
overcloud-controller-1
f12e669017cf
heat
b0c08a9f69da
root
e11613aad35b
nova
ec331f5ce148
keystone
Comment 5 Omri Hochman 2017-11-01 10:02:02 EDT
vi /var/spool/cron/keystone

# HEADER: This file was autogenerated at 2017-10-31 18:01:52 +0000 by puppet.
# HEADER: While it can still be managed manually, it is definitely not recommended.
# HEADER: Note particularly that the comments starting with 'Puppet Name' should
# HEADER: not be deleted, as doing so could cause duplicate cron jobs.
# Puppet Name: cinder-manage db purge
PATH=/bin:/usr/bin:/usr/sbin SHELL=/bin/sh
1 0 * * * cinder-manage db purge 0 >>/var/log/cinder/cinder-rowsflush.log 2>&1
Comment 6 Dan Prince 2017-11-01 14:46:31 EDT
This fails downstream still because there were two related upstream patches to fix the issue. In the first patch we missed added the Ansible tags so that the new task runs at step 2. See: https://review.openstack.org/#/c/497936/

I added an external tracker for this and will double check to make sure it gets backported.
Comment 7 Dan Prince 2017-11-01 15:05:30 EDT
Submitted the patch upstream for backporting to Pike:

https://review.openstack.org/517082
Comment 8 Martin André 2017-11-10 05:04:58 EST
https://review.openstack.org/517082 was merged in stable/pike.
Comment 10 Artem Hrechanychenko 2017-11-15 10:05:59 EST
VERIFIED

openstack-tripleo-heat-templates-7.0.3-3.el7ost.noarch

undercloud) [stack@undercloud-0 ~]$ for i in `openstack server list -f value -c Networks |awk -F'=' '{print $2}'`; do echo "############################################"; ssh -o StrictHostKeyChecking=no heat-admin@$i "hostname; sudo ls /var/spool/cron/"; done
############################################
overcloud-controller-2
cinder
############################################
overcloud-controller-0
cinder
############################################
overcloud-controller-1
cinder
############################################
overcloud-compute-0
############################################
Warning: Permanently added '192.168.24.11' (ECDSA) to the list of known hosts.
overcloud-compute-1


(undercloud) [stack@undercloud-0 ~]$ for i in `openstack server list -f value -c Networks |awk -F'=' '{print $2}'`; do echo "############################################"; ssh -o StrictHostKeyChecking=no heat-admin@$i "hostname; sudo docker ps|grep cron"; done
############################################
overcloud-controller-2
d947707d1a33        192.168.24.1:8787/rhosp12/openstack-heat-api-docker:2017-11-14.4                  "kolla_start"            12 hours ago        Up 12 hours (healthy)                       heat_api_cron
9dcbafd658b6        192.168.24.1:8787/rhosp12/openstack-cron-docker:2017-11-14.4                      "kolla_start"            12 hours ago        Up 12 hours                                 logrotate_crond
ed8146918768        192.168.24.1:8787/rhosp12/openstack-nova-api-docker:2017-11-14.4                  "kolla_start"            13 hours ago        Up 13 hours (healthy)                       nova_api_cron
a46a64801e4f        192.168.24.1:8787/rhosp12/openstack-keystone-docker:2017-11-14.4                  "/bin/bash -c '/usr/l"   13 hours ago        Up 13 hours (healthy)                       keystone_cron
############################################
overcloud-controller-0
b8ae251e37f3        192.168.24.1:8787/rhosp12/openstack-heat-api-docker:2017-11-14.4                  "kolla_start"            12 hours ago        Up 12 hours (healthy)                       heat_api_cron
8449f9fde865        192.168.24.1:8787/rhosp12/openstack-cron-docker:2017-11-14.4                      "kolla_start"            12 hours ago        Up 12 hours                                 logrotate_crond
5782fc9098a8        192.168.24.1:8787/rhosp12/openstack-nova-api-docker:2017-11-14.4                  "kolla_start"            12 hours ago        Up 12 hours (healthy)                       nova_api_cron
b9d41546e865        192.168.24.1:8787/rhosp12/openstack-keystone-docker:2017-11-14.4                  "/bin/bash -c '/usr/l"   13 hours ago        Up 13 hours (healthy)                       keystone_cron
############################################
overcloud-controller-1
19f0dde2ac00        192.168.24.1:8787/rhosp12/openstack-heat-api-docker:2017-11-14.4                  "kolla_start"            12 hours ago        Up 12 hours (healthy)                       heat_api_cron
8ae65be2631c        192.168.24.1:8787/rhosp12/openstack-cron-docker:2017-11-14.4                      "kolla_start"            12 hours ago        Up 12 hours                                 logrotate_crond
1371f7b0dffd        192.168.24.1:8787/rhosp12/openstack-nova-api-docker:2017-11-14.4                  "kolla_start"            13 hours ago        Up 13 hours (healthy)                       nova_api_cron
edf1489ac5e8        192.168.24.1:8787/rhosp12/openstack-keystone-docker:2017-11-14.4                  "/bin/bash -c '/usr/l"   13 hours ago        Up 13 hours (healthy)                       keystone_cron
############################################
overcloud-compute-0
aa4c78b7ee4d        192.168.24.1:8787/rhosp12/openstack-cron-docker:2017-11-14.4                 "kolla_start"       13 hours ago        Up 17 minutes                                   logrotate_crond
############################################
overcloud-compute-1
bdefb26d499a        192.168.24.1:8787/rhosp12/openstack-cron-docker:2017-11-14.4                 "kolla_start"       13 hours ago        Up 13 hours                                   logrotate_crond
Comment 13 errata-xmlrpc 2017-12-13 16:51:30 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462

Note You need to log in before you can comment on or make changes to this bug.