This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1476772 - OSP11 -> OSP12: after upgrade the Openstack related cron jobs are running as commands on the host and not inside containers
OSP11 -> OSP12: after upgrade the Openstack related cron jobs are running as ...
Status: ON_QA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates (Show other bugs)
12.0 (Pike)
Unspecified Unspecified
urgent Severity urgent
: ga
: 12.0 (Pike)
Assigned To: Emilien Macchi
Gurenko Alex
: Triaged
Depends On:
Blocks: 1399762
  Show dependency treegraph
 
Reported: 2017-07-31 08:24 EDT by Marius Cornea
Modified: 2017-10-16 12:11 EDT (History)
14 users (show)

See Also:
Fixed In Version: openstack-tripleo-heat-templates-7.0.0-0.20170805163048.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1701254 None None None 2017-07-31 09:34 EDT
OpenStack gerrit 478915 None None None 2017-08-01 03:17 EDT
OpenStack gerrit 478919 None None None 2017-08-01 03:17 EDT
OpenStack gerrit 485858 None None None 2017-08-01 03:19 EDT
OpenStack gerrit 485863 None None None 2017-08-01 03:18 EDT
OpenStack gerrit 485865 None None None 2017-08-01 03:18 EDT
OpenStack gerrit 485873 None None None 2017-08-01 03:19 EDT

  None (edit)
Description Marius Cornea 2017-07-31 08:24:39 EDT
Description of problem:
After upgrading from OSP11 to the OSP12 containerized architecture the cron jobs used by keystone, nova are still running as commands on the host. I'd expect them to be run inside the newly created containers.

Cron jobs:

[root@controller-0 cron]# ls -l /var/spool/cron/
total 16
-rw-------. 1 root root 476 Jul 31 11:43 ceilometer
-rw-------. 1 root root 494 Jul 31 11:49 heat
-rw-------. 1 root root 676 Jul 31 11:50 keystone
-rw-------. 1 root root 510 Jul 31 11:52 nova

[root@controller-0 cron]# cat /var/spool/cron/heat 
# HEADER: This file was autogenerated at 2017-07-31 11:49:24 +0000 by puppet.
# HEADER: While it can still be managed manually, it is definitely not recommended.
# HEADER: Note particularly that the comments starting with 'Puppet Name' should
# HEADER: not be deleted, as doing so could cause duplicate cron jobs.
# Puppet Name: heat-manage purge_deleted
PATH=/bin:/usr/bin:/usr/sbin SHELL=/bin/sh
1 0 * * * sleep `expr ${RANDOM} \% 3600`; heat-manage purge_deleted -g days 30 >>/dev/null 2>&1

[root@controller-0 cron]# cat /var/spool/cron/keystone 
# HEADER: This file was autogenerated at 2017-07-31 11:50:18 +0000 by puppet.
# HEADER: While it can still be managed manually, it is definitely not recommended.
# HEADER: Note particularly that the comments starting with 'Puppet Name' should
# HEADER: not be deleted, as doing so could cause duplicate cron jobs.
# Puppet Name: keystone-manage token_flush
PATH=/bin:/usr/bin:/usr/sbin SHELL=/bin/sh
1 * * * * sleep `expr ${RANDOM} \% 0`; keystone-manage token_flush >>/var/log/keystone/keystone-tokenflush.log 2>&1
# Puppet Name: cinder-manage db purge
PATH=/bin:/usr/bin:/usr/sbin SHELL=/bin/sh
1 0 * * * cinder-manage db purge 0 >>/var/log/cinder/cinder-rowsflush.log 2>&1

[root@controller-0 cron]# cat /var/spool/cron/nova 
# HEADER: This file was autogenerated at 2017-07-31 11:52:44 +0000 by puppet.
# HEADER: While it can still be managed manually, it is definitely not recommended.
# HEADER: Note particularly that the comments starting with 'Puppet Name' should
# HEADER: not be deleted, as doing so could cause duplicate cron jobs.
# Puppet Name: nova-manage db archive_deleted_rows
PATH=/bin:/usr/bin:/usr/sbin SHELL=/bin/sh
1 0 * * * nova-manage db archive_deleted_rows --max_rows 100  >>/var/log/nova/nova-rowsflush.log 2>&1
 

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-7.0.0-0.20170721174554.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP11
2. Upgrade to OSP12
3. Check Openstack related cron jobs set up by OSPd

Actual results:
The cron jobs are running as regular commands on host

Expected results:
The cron jobs are running inside the containers created during upgrade.

Additional info:
Comment 2 Marius Cornea 2017-08-01 03:35:10 EDT
It looks like we're missing the cron jobs removal during upgrade. Currently, with the latest puddle we have the nova_api_cron container running but the cron job still exists on the host:

[root@controller-1 heat-admin]# docker ps | grep cron
053adcc4b586        192.168.24.1:8787/rhosp12/openstack-nova-api-docker:2017-07-26.10                  "/usr/sbin/crond -n"     14 hours ago        Up 14 hours                                       nova_api_cron

[root@controller-1 heat-admin]# ls -l /var/spool/cron/
total 12
-rw-------. 1 root root 494 Jul 31 15:30 heat
-rw-------. 1 root root 676 Jul 31 15:31 keystone
-rw-------. 1 root root 510 Jul 31 15:32 nova
Comment 3 Marius Cornea 2017-08-01 03:41:08 EDT
How are the cron containers running the cron jobs? I checked /var/spool/cron/ inside containers and it's empty:

[root@controller-1 heat-admin]# docker exec -it nova_api_cron bash -c 'ls /var/spool/cron/'
Comment 4 Ollie Walsh 2017-08-01 06:06:50 EDT
Should be resolved by https://review.openstack.org/485858

Note You need to log in before you can comment on or make changes to this bug.