Description of problem: Provide admin the ability to set a Swift endpoint as a valid backup destination for the CFME database. Version-Release number of selected component (if applicable): Running on CloudForms 4.5 at this time. CFME Version: 5.8.2.3.20171016155816_aaec796 How reproducible: Currently only CIFS/NFS are supported as backup destinations. Amazon S3 has just been added under a separate RFE. Steps to Reproduce: 1. Setup CFME Backup to External Storage Actual results: No Swift endpoint feasible. Expected results: Ideally add in the ability to set a URI, user, and key to push backups directly into a bucket. Additional info: RHN Case: 01974285
Information imported from https://bugzilla.redhat.com/show_bug.cgi?id=1513520
PRs added so far https://github.com/ManageIQ/manageiq-schema/pull/259 for schema changes https://github.com/ManageIQ/manageiq-gems-pending for backup side of Mount Session
https://github.com/ManageIQ/manageiq/pull/17967 for core ManageIQ changes.
https://github.com/ManageIQ/manageiq-ui-classic/pull/4684 for UI changes. Schema PR above is merged. More to come.
Appliance console PR is https://github.com/ManageIQ/manageiq-appliance_console/pull/66
UI PR is merged. Core PR is merged. Still pending are gems-pending and appliance_console.
New commit detected on ManageIQ/manageiq/hammer: https://github.com/ManageIQ/manageiq/commit/e5e724a18e9887514fc0753b7d122991c9f81ec4 commit e5e724a18e9887514fc0753b7d122991c9f81ec4 Author: Nick Carboni <ncarboni> AuthorDate: Thu Oct 11 11:48:16 2018 -0400 Commit: Nick Carboni <ncarboni> CommitDate: Thu Oct 11 11:48:16 2018 -0400 Merge pull request #17967 from jerryk55/swift_db_backups Openstack Swift DB Backups (cherry picked from commit 520f0077af3b3373bab9b692f50224575cfc332b) https://bugzilla.redhat.com/show_bug.cgi?id=1615488 app/models/database_backup.rb | 7 +- app/models/file_depot.rb | 4 + app/models/file_depot_swift.rb | 67 + app/models/miq_schedule.rb | 20 +- app/models/mixins/file_depot_mixin.rb | 7 +- lib/evm_database_ops.rb | 7 +- locale/en.yml | 1 + spec/models/file_depot_spec.rb | 1 + spec/models/file_depot_swift_spec.rb | 29 + 9 files changed, 129 insertions(+), 14 deletions(-)
gems-pending (backup side) is merged. Still pending is appliance_console, and work to fix restore (all storage types) by Nick LaMuro.
New commit detected on ManageIQ/manageiq-gems-pending/hammer: https://github.com/ManageIQ/manageiq-gems-pending/commit/66584e4fdeac5863860591bb4eb34c1fe34ecd50 commit 66584e4fdeac5863860591bb4eb34c1fe34ecd50 Author: Nick Carboni <ncarboni> AuthorDate: Wed Oct 17 11:32:28 2018 -0400 Commit: Nick Carboni <ncarboni> CommitDate: Wed Oct 17 11:32:28 2018 -0400 Merge pull request #371 from jerryk55/swift_db_backup DB Backups to Openstack Swift (cherry picked from commit 33ec84a811d119f6c605385c2164b53af8ab678e) https://bugzilla.redhat.com/show_bug.cgi?id=1615488 lib/gems/pending/util/miq_object_storage.rb | 33 + lib/gems/pending/util/object_storage/miq_swift_storage.rb | 159 + manageiq-gems-pending.gemspec | 2 + spec/support/contexts/generated_tmp_files.rb | 13 +- spec/util/miq_file_storage_spec.rb | 24 + spec/util/miq_object_storage_spec.rb | 120 + spec/util/object_storage/miq_swift_storage_spec.rb | 69 + 7 files changed, 419 insertions(+), 1 deletion(-)
https://github.com/ManageIQ/manageiq-appliance/pull/215
New commit detected on ManageIQ/manageiq-appliance/master: https://github.com/ManageIQ/manageiq-appliance/commit/9faaeedfd32b8456774519e6f09a156d852d8e6e commit 9faaeedfd32b8456774519e6f09a156d852d8e6e Author: Nick Carboni <ncarboni> AuthorDate: Mon Nov 5 14:14:02 2018 -0500 Commit: Nick Carboni <ncarboni> CommitDate: Mon Nov 5 14:14:02 2018 -0500 Add console version 3.3 Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1586176 Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1586187 Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1615488 Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1632433 manageiq-appliance-dependencies.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
New commit detected on ManageIQ/manageiq-appliance/hammer: https://github.com/ManageIQ/manageiq-appliance/commit/b6b244d6b71cd26babca371c969e2447f1799ffa commit b6b244d6b71cd26babca371c969e2447f1799ffa Author: Brandon Dunne <brandondunne> AuthorDate: Tue Nov 6 10:45:27 2018 -0500 Commit: Brandon Dunne <brandondunne> CommitDate: Tue Nov 6 10:45:27 2018 -0500 Merge pull request #215 from carbonin/use_version_3_3_of_the_console Add console version 3.3 (cherry picked from commit ce975b028a06a43c4edc04fadf5a895015b4e874) Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1586176 Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1586187 Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1615488 Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1632433 manageiq-appliance-dependencies.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Being able to set OpenStack Swift as a backup target is not working for me with build 5.10.0.27.20181128170555_43ed8cb Selecting Type OpenStack Swift when adding a new schedule does not show the fields needed for Swift. This also occurs when trying to run an adhoc backup. I have tried with both Firefox (version 63) and Chrome (version 71)
Brant, can you provide access to an appliance exhibiting this issue? My personal appliance running from master works as expected.
Yup. Looks like https://github.com/ManageIQ/manageiq-ui-classic/pull/4684 is not on that appliance. Satoe, is this build supposed to contain this PR? Thanks much. Jerry
Apologies in advance since I was not keeping up with the dates of any of these releases - but I thought that this would have been long before back ports for Hammer - October 11? I could be 100% mistaken of course. From my cloudy memory this made it in time to get into the release.
Adding the blocker flag, even though this is late in the cycle, as OSP Swift was part of the RFE work list
New commit detected on ManageIQ/manageiq-ui-classic/hammer: https://github.com/ManageIQ/manageiq-ui-classic/commit/c09421afa2a460ab63668a5df89d22c7ea0743e0 commit c09421afa2a460ab63668a5df89d22c7ea0743e0 Author: Harpreet Kataria <hkataria> AuthorDate: Thu Oct 11 12:22:32 2018 -0400 Commit: Harpreet Kataria <hkataria> CommitDate: Thu Oct 11 12:22:32 2018 -0400 Merge pull request #4684 from jerryk55/db_backups_swift Swift DB Backups (cherry picked from commit a2abbc78866bd69665f3fc93a298bf0b4c7bec8d) https://bugzilla.redhat.com/show_bug.cgi?id=1615488 app/assets/javascripts/controllers/ops/diagnostics_database_form_controller.js | 41 +- app/assets/javascripts/controllers/schedule/schedule_form_controller.js | 34 +- app/assets/javascripts/services/miq_db_backup_service.js | 18 +- app/controllers/ops_controller/diagnostics.rb | 27 +- app/controllers/ops_controller/settings/schedules.rb | 70 +- app/views/ops/_diagnostics_database_tab.html.haml | 76 + app/views/ops/_schedule_form_filter.html.haml | 79 +- 7 files changed, 306 insertions(+), 39 deletions(-)
Unable to verify in 5.10.0.32.20190115185124_c957ada, @jkeselma and I both were unsuccessful in the verification. Pushing this back to ON_DEV. Steps taken to verify: 1) Navigate to "Configuration" 2) Click on Schedules 3) Add a new schedule 4) Enter Name, Description, Action -> "Database Backup", Type -> "OpenStack Swift" 5) the following fields depend on your OpenStack server but I tested on RHOS11 and RHOS13 without success 6) API version -> "Keystone v3", Domain ID -> "default" 7) Enter any Depot Name 8) For the URI, I entered the IP address of the the OpenStack instance + the authentication port (default of 5000) 9) The region, security protocol, and creds depend on your OpenStack server 10) For API port, it is unclear whether this is the port used for authentication or the port that is used for swift's Object Storage. I noticed that whatever you put here will be concatenated onto the URI. If API port is meant to be the port used for Swift's Object Storage, this behavior is undesirable and should be another BZ. I tried both with no success. In the case of using the Object Storage port, I was unable to access the swift container with 400 Bad Request. When I used the the authentication port, I was unable to access the swift container with a 401 Unauthorized error. 11) Setup the schedule to run. When the schedule runs, I got "Error getting Swift Container", with 401 and 400 errors. @jkeselma however received 404 errors when he tried to verify.
After working with @jkeselma this morning, we were both able to verify this with both SSL and Non-SSL connections in CFME 5.10.0.32.20190115185124_c957ada Steps taken to verify: 1) Navigate to "Configuration" 2) Click on Schedules 3) Add a new schedule 4) Enter Name, Description, Action -> "Database Backup", Type -> "OpenStack Swift" 5) the following fields depend on your OpenStack server 6) API version -> "Keystone v3", Domain ID -> "default" 7) Enter any Depot Name 8) For the URI, I entered the IP address of the the OpenStack instance + the authentication port + any name for the folder in which to store the backup (e.g. "10.x.x.x:<authentication_port>/jdupuy-db-backup1") 9) The region, security protocol, and creds depend on your OpenStack server 10) For API port, put the authentication port for the server 11) Click validate on creds -> credentials should be validated 12) Setup the schedule to run Received the following output from the EVM log: [----] D, [2019-01-23T10:05:19.633326 #14301:113ef60] DEBUG -- : Swift container jdupuy-test1-db does not exist. Creating. [----] D, [2019-01-23T10:05:19.810112 #14301:113ef60] DEBUG -- : Swift container [jdupuy-test1-db] created [----] D, [2019-01-23T10:05:23.571237 #14310:113ef60] DEBUG -- : Swift container [#<Fog::Storage::OpenStack::Directory:0x000000001232c820>] found [----] I, [2019-01-23T10:06:02.063751 #14301:113ef60] INFO -- : MIQ(EvmDatabaseOps.backup) [vmdb_production] database has been backed up to file: [swift://10.x.x.x:13000/jdupuy-test1-db/db_backup/region_0/jdupuy_test1_swfit/region_0_20190123_150512.backup] And the task was marked as successful in the UI.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2019:0212