Red Hat Bugzilla – Bug 1075609
openstack-cinder-backup service is not running after install
Last modified: 2017-10-18 11:43:42 EDT
Description of problem: openstack-cinder-backup should be started at the end of the installation; it is enabled in chkconfig though Version-Release number of selected component (if applicable): openstack-packstack-2013.2.1-0.28.dev989.el6ost.noarch
Startup of the service is implemented in Packstack, so are there any errors in Cinder logs? Can you attach <ip>_cinder.pp.log file?
Created attachment 875423 [details] cinder_backup_trace.txt hi Martin, there is a trace in the backup log indeed, thanks! Apparently a problem with the database layout. .pp seems to have been applied correctly, I'm attaching it though. I don't think packstack has any control over the "db sync" operation so maybe this should be reassigned to openstack-cinder ?
Created attachment 875424 [details] cinder.pp.log
Hi Giulio, I cannot reproduce it: - RHEL6.5.z - Latest RHOS (as of today) against openstack-packstack-2013.2.1-0.28.dev989.el6ost.noarch Could you please provide answer file? So I can test against it? Regards, Gilles
Also, the cinder/backup.log error is specific to cinder while doing a volume backup. It sounds like a cinder issue, passing it them. That said, I'm happy to run again the test with you answer file.
Created attachment 878315 [details] cinder backup logs
Just to clarify for others getting here. 1. From a configuration viewpoint, Cinder Backup is activated only if Swift option is selected: https://github.com/stackforge/packstack/blob/master/packstack/plugins/cinder_250.py#L422 Which makes sense according to http://docs.openstack.org/grizzly/openstack-block-storage/admin/content/block_storage_overview.html: << cinder-backup - The cinder-backup service provides a means to back up a Cinder Volume to OpenStack Object Store (SWIFT). >> Tests are showing this behavior: - If Cinder option is 'yes' but Swift is 'no': openstack-cinder-backup 0:off 1:off 2:off 3:off 4:off 5:off 6:off - If Cinder is selected as well as Swift: openstack-cinder-backup 0:off 1:off 2:on 3:on 4:on 5:on 6:off 2. I cannot reproduce the issue, having Cinder and Swift selected, Cinder backup is activated and started without errors see attachment See above ttachment 878315
Hi Giulio, I'm able to reproduce it. Using the following cmdline: packstack --allinone --nagios-install=n --os-ceilometer-install=n --os-neutron-install=n --keystone-admin-passwd=blah --keystone-demo-passwd=blah --ssh-public-key=/root/.ssh/id_rsa.pub At the end of the installation, /var/log/cinder/backup.log is showing the same error: 2014-04-03 01:53:56.905 6958 TRACE cinder.service ProgrammingError: (ProgrammingError) (1146, "Table 'cinder.services' doesn't exist") 'SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.host AS services_host, services.`binary` AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.availability_zone AS services_availability_zone \nFROM services \nWHERE services.deleted = %s AND services.host = %s AND services.`binary` = %s \n LIMIT %s' (0, 'rhel65-t1.surfgate.org', 'cinder-backup', 1) The interesting thing is after restarting the service, everything is back to normal (which more likely why I missed in in the first place), which would be a work around for everyone but CI user (jenkins/gerribot), unless adding a cmd somewhere: [root@rhel65-t1 ~]# service openstack-cinder-backup status openstack-cinder-backup dead but pid file exists [root@rhel65-t1 ~]# service openstack-cinder-backup restart Stopping openstack-cinder-backup: [FAILED] Starting openstack-cinder-backup: [ OK ] [root@rhel65-t1 ~]# service openstack-cinder-backup status openstack-cinder-backup (pid 15980) is running... [root@rhel65-t1 ~]# tail -f /var/log/cinder/backup.log 2014-04-03 01:53:56.905 6958 TRACE cinder.service 2014-04-03 01:53:56.935 6933 INFO cinder.service [-] Child 6958 exited with status 2 2014-04-03 01:53:56.935 6933 INFO cinder.service [-] _wait_child 1 2014-04-03 01:53:56.935 6933 INFO cinder.service [-] wait wrap.failed True 2014-04-03 02:40:04.132 15980 INFO cinder.service [-] Starting 1 workers 2014-04-03 02:40:04.133 15980 INFO cinder.service [-] Started child 15988 2014-04-03 02:40:04.136 15988 AUDIT cinder.service [-] Starting cinder-backup node (version 2013.2.2) 2014-04-03 02:40:04.367 15988 INFO cinder.openstack.common.rpc.impl_qpid [req-91b8db79-cb5d-4714-a526-e8207fa7427f None None] Connected to AMQP server on 192.168.22.2:5672 2014-04-03 02:40:04.376 15988 INFO cinder.backup.manager [req-91b8db79-cb5d-4714-a526-e8207fa7427f None None] Starting volume driver LVMISCSIDriver (2.0.0). 2014-04-03 02:40:04.741 15988 INFO cinder.backup.manager [req-91b8db79-cb5d-4714-a526-e8207fa7427f None None] Cleaning up incomplete backup operations.
There is no service dependency created for cinder classes, therefore the result is aleatory and cinder-backup might starts before cinder-manage db_sync has been executed! Notice: /Stage[main]/Cinder::Api/Service[cinder-api]/ensure: ensure changed 'stopped' to 'running' Notice: /Stage[main]/Cinder::Backup/Service[cinder-backup]/ensure: ensure changed 'stopped' to 'running' Notice: /Stage[main]/Cinder::Api/Exec[cinder-manage db_sync]: Triggered 'refresh' from 36 events Notice: /Stage[main]/Cinder::Volume/Service[cinder-volume]/ensure: ensure changed 'stopped' to 'running' Notice: /Stage[main]/Cinder::Volume::Iscsi/Service[tgtd]/ensure: ensure changed 'stopped' to 'running' Notice: /Stage[main]/Cinder::Scheduler/Service[cinder-scheduler]/ensure: ensure changed 'stopped' to 'running' Testing a fix using resource collector (Stolen from Quickstack): Class['swift'] -> Service <| |>
The fix has been submitted upstream and is waiting for review: https://review.openstack.org/#/c/85910/
https://review.openstack.org/#/c/85910/ has been merged.
creating backport to havana
verified in openstack-puppet-modules-2013.2-9.2.el6ost.noarch openstack-packstack-2013.2.1-0.32.dev1040.el6ost.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2014-1691.html