Bug 1476772
Summary: | OSP11 -> OSP12: after upgrade the Openstack related cron jobs are running as commands on the host and not inside containers | ||
---|---|---|---|
Product: | Red Hat OpenStack | Reporter: | Marius Cornea <mcornea> |
Component: | openstack-tripleo-heat-templates | Assignee: | Carlos Camacho <ccamacho> |
Status: | CLOSED ERRATA | QA Contact: | Marius Cornea <mcornea> |
Severity: | urgent | Docs Contact: | |
Priority: | urgent | ||
Version: | 12.0 (Pike) | CC: | ccamacho, dbecker, jcoufal, jstransk, mandreou, mbracho, mbultel, mburns, mcornea, morazi, ohochman, owalsh, rhel-osp-director-maint, sasha, sathlang, sclewis, tvignaud |
Target Milestone: | beta | Keywords: | TestOnly, Triaged |
Target Release: | 12.0 (Pike) | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | openstack-tripleo-heat-templates-7.0.0-0.20170805163048.el7ost | Doc Type: | If docs needed, set a value |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2017-12-13 21:45:43 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | |||
Bug Blocks: | 1399762 |
Description
Marius Cornea
2017-07-31 12:24:39 UTC
It looks like we're missing the cron jobs removal during upgrade. Currently, with the latest puddle we have the nova_api_cron container running but the cron job still exists on the host: [root@controller-1 heat-admin]# docker ps | grep cron 053adcc4b586 192.168.24.1:8787/rhosp12/openstack-nova-api-docker:2017-07-26.10 "/usr/sbin/crond -n" 14 hours ago Up 14 hours nova_api_cron [root@controller-1 heat-admin]# ls -l /var/spool/cron/ total 12 -rw-------. 1 root root 494 Jul 31 15:30 heat -rw-------. 1 root root 676 Jul 31 15:31 keystone -rw-------. 1 root root 510 Jul 31 15:32 nova How are the cron containers running the cron jobs? I checked /var/spool/cron/ inside containers and it's empty: [root@controller-1 heat-admin]# docker exec -it nova_api_cron bash -c 'ls /var/spool/cron/' Should be resolved by https://review.openstack.org/485858 After upgrade cron jobs are running both on the host and inside containers. I believe we're missing removing the ones running on the host during upgrade: [root@controller-0 ~]# ls -l /var/spool/cron/ total 12 -rw-------. 1 root root 494 Nov 8 12:54 heat -rw-------. 1 root root 647 Nov 8 12:54 keystone -rw-------. 1 root root 510 Nov 8 12:58 nova [root@controller-0 ~]# docker ps | grep cron 4bb5f4a42888 rhos-qe-mirror-tlv.usersys.redhat.com:5000/rhosp12/openstack-heat-api-docker:20171103.1 "kolla_start" 11 minutes ago Up 11 minutes (healthy) heat_api_cron ea63fc2ded3a rhos-qe-mirror-tlv.usersys.redhat.com:5000/rhosp12/openstack-cron-docker:20171103.1 "kolla_start" 11 minutes ago Up 11 minutes logrotate_crond a01afae02e7c rhos-qe-mirror-tlv.usersys.redhat.com:5000/rhosp12/openstack-nova-api-docker:20171103.1 "kolla_start" 11 minutes ago Up 11 minutes (healthy) nova_api_cron 93ded582b6d8 rhos-qe-mirror-tlv.usersys.redhat.com:5000/rhosp12/openstack-keystone-docker:20171103.1 "/bin/bash -c '/usr/l" 14 minutes ago Up 14 minutes (healthy) keystone_cron [root@controller-0 ~]# [root@controller-0 ~]# grep autogenerated /var/spool/cron/nova # HEADER: This file was autogenerated at 2017-11-08 12:58:19 +0000 by puppet. [root@controller-0 ~]# docker exec -it nova_api_cron grep autogenerated /var/spool/cron/nova # HEADER: This file was autogenerated at 2017-11-08 16:15:23 +0000 by puppet. removing untriaged to be discuess with the team Carlos, Can you take a look to see if there is remaining cleanup is needed? Hey Mike, checking the code, this bug should be already fixed. The cron cleanup was merged with this review https://review.openstack.org/#/c/490496/ The following cron files were cleaned on the upgrade tasks in: docker/services/cinder-api.yaml docker/services/heat-api.yaml docker/services/keystone.yaml docker/services/nova-api.yaml Ceilometer services were removed. Marius, can you confirm that the fix was tested? (In reply to Carlos Camacho from comment #14) > Hey Mike, checking the code, this bug should be already fixed. > > The cron cleanup was merged with this review > https://review.openstack.org/#/c/490496/ > > The following cron files were cleaned on the upgrade tasks in: > > docker/services/cinder-api.yaml > docker/services/heat-api.yaml > docker/services/keystone.yaml > docker/services/nova-api.yaml > > > Ceilometer services were removed. > > > > Marius, can you confirm that the fix was tested? https://review.openstack.org/#/c/490496/ has been merged for some time(in August) so it was part of the build used for testing at the time of comment #10 This issue got addressed by https://review.openstack.org/#/c/517082/ which is present in the latest puddle build. [root@controller-0 heat-admin]# ls -l /var/spool/cron/ total 4 -rw-------. 1 root root 474 Nov 21 22:04 cinder [root@controller-0 heat-admin]# cd [root@controller-0 ~]# ls -l /var/spool/cron/ total 4 -rw-------. 1 root root 474 Nov 21 22:04 cinder [root@controller-0 ~]# docker ps | grep cron 4ec0c3ac4038 rhos-qe-mirror-brq.usersys.redhat.com:5000/rhosp12/openstack-heat-api-docker:20171121.1 "kolla_start" 31 minutes ago Up 31 minutes (healthy) heat_api_cron 94301582305f rhos-qe-mirror-brq.usersys.redhat.com:5000/rhosp12/openstack-cron-docker:20171121.1 "kolla_start" 31 minutes ago Up 31 minutes logrotate_crond 7a8073813cc4 rhos-qe-mirror-brq.usersys.redhat.com:5000/rhosp12/openstack-nova-api-docker:20171121.1 "kolla_start" 31 minutes ago Up 31 minutes (healthy) nova_api_cron fbda5319036e rhos-qe-mirror-brq.usersys.redhat.com:5000/rhosp12/openstack-keystone-docker:20171121.1 "/bin/bash -c '/usr/l" 34 minutes ago Up 34 minutes (healthy) keystone_cron [root@controller-0 ~]# docker exec nova_api_cron ls -l /var/spool/cron/ total 4 -rw-------. 1 root root 510 Nov 22 13:23 nova [root@controller-0 ~]# docker exec heat_api_cron ls -l /var/spool/cron/ total 4 -rw-------. 1 root root 494 Nov 22 13:23 heat [root@controller-0 ~]# docker exec keystone_cron ls -l /var/spool/cron/ total 4 -rw-------. 1 root root 487 Nov 22 13:23 keystone Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3462 |