Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
When offline backup of Satellite server fails for any reason, it leaves the Satellite services in stopped status, and the maintenance mode enabled, which makes Satellite services unavailable till the following actions are manually performed:
1. Start Satellite services:
~~~
# foreman-maintain service start
~~~
2. Disable maintenance mode:
~~~
# foreman-maintain maintenance-mode stop
~~~
Would be more efficient, and less impacting to a customer running into this situation, to have the above 2 actions done by the backup feature when it detects a failure.
Version-Release number of selected component (if applicable):
satellite-maintain-0.0.1-1.el7sat.noarch
How reproducible:
All the time reproduceable.
Steps to Reproduce:
1. Launch an offline backup of Satellite server:
~~~
# foreman-maintain backup offline --assumeyes /backups
~~~
2. Press CTRL + C in the middle of the backup operation to interrupt it:
~~~
# foreman-maintain backup offline --assumeyes /backups
Starting backup: 2021-03-04 09:07:47 +0000
Running preparation steps required to run the next scenarios
================================================================================
Make sure Foreman DB is up:
/ Checking connection to the Foreman DB [OK]
--------------------------------------------------------------------------------
Running Backup
================================================================================
Confirm turning off services is allowed:
WARNING: This script will stop your services.
Do you want to proceed? (assuming yes)
[OK]
--------------------------------------------------------------------------------
Prepare backup Directory:
Creating backup folder /backups/satellite-backup-2021-03-04-09-07-47 [OK]
--------------------------------------------------------------------------------
Check if the directory exists and is writable: [OK]
--------------------------------------------------------------------------------
Generate metadata:
/ Saving metadata to metadata.yml [OK]
--------------------------------------------------------------------------------
Detect features available in the local proxy: [OK]
--------------------------------------------------------------------------------
disable active sync plans:
\ Total 0 sync plans are now disabled. [OK]
--------------------------------------------------------------------------------
Add maintenance_mode chain to iptables: [OK]
--------------------------------------------------------------------------------
Stop applicable services:
Stopping the following service(s):
rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, rh-redis5-redis, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflow-sidekiq@orchestrator, httpd, puppetserver, dynflow-sidekiq@worker, dynflow-sidekiq@worker-hosts-queue, foreman-proxy
- All services stopped [OK]
--------------------------------------------------------------------------------
Backup config files:
| Collecting config files to backup [OK]
--------------------------------------------------------------------------------
Backup Pulp data:
/ Collecting Pulp data^C [RUNNING]
--------------------------------------------------------------------------------
/usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/utils/command_runner.rb:88:in `read': Interrupt
from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/utils/command_runner.rb:88:in `block in run_non_interactively'
from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/utils/command_runner.rb:83:in `popen'
from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/utils/command_runner.rb:83:in `run_non_interactively'
from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/utils/command_runner.rb:30:in `run'
from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/concerns/system_helpers.rb:59:in `execute_runner'
from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/concerns/system_helpers.rb:64:in `execute!'
from /usr/share/gems/gems/foreman_maintain-0.6.15/definitions/features/tar.rb:26:in `run'
from /usr/share/gems/gems/foreman_maintain-0.6.15/definitions/procedures/backup/pulp.rb:30:in `pulp_backup'
from /usr/share/gems/gems/foreman_maintain-0.6.15/definitions/procedures/backup/pulp.rb:21:in `block (2 levels) in run'
from /usr/share/ruby/fileutils.rb:125:in `chdir'
from /usr/share/ruby/fileutils.rb:125:in `cd'
from /usr/share/gems/gems/foreman_maintain-0.6.15/definitions/procedures/backup/pulp.rb:17:in `block in run'
from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/reporter/cli_reporter.rb:141:in `with_spinner'
from /usr/share/gems/gems/foreman_maintain-0.6.15/definitions/procedures/backup/pulp.rb:16:in `run'
from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/executable.rb:139:in `__run__'
from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/runner/execution.rb:83:in `block (2 levels) in run'
from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/runner/execution.rb:107:in `capture_errors'
from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/runner/execution.rb:82:in `block in run'
from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/runner/execution.rb:101:in `with_metadata_calculation'
from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/runner/execution.rb:81:in `run'
from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/runner.rb:123:in `run_step'
from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/runner.rb:112:in `run_steps'
from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/runner.rb:105:in `execute_scenario_steps'
from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/runner.rb:51:in `run_scenario'
from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/runner.rb:33:in `block in run'
from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/runner.rb:32:in `each'
from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/runner.rb:32:in `run'
from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/cli/base.rb:64:in `run_scenario'
from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/cli/backup_command.rb:25:in `perform_backup'
from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/cli/backup_command.rb:125:in `execute'
from /usr/share/gems/gems/clamp-0.6.2/lib/clamp/command.rb:67:in `run'
from /usr/share/gems/gems/clamp-0.6.2/lib/clamp/subcommand/execution.rb:11:in `execute'
from /usr/share/gems/gems/clamp-0.6.2/lib/clamp/command.rb:67:in `run'
from /usr/share/gems/gems/clamp-0.6.2/lib/clamp/subcommand/execution.rb:11:in `execute'
from /usr/share/gems/gems/clamp-0.6.2/lib/clamp/command.rb:67:in `run'
from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/cli.rb:34:in `run'
from /usr/share/gems/gems/clamp-0.6.2/lib/clamp/command.rb:125:in `run'
from /usr/share/gems/gems/foreman_maintain-0.6.15/bin/foreman-maintain:15:in `<top (required)>'
from /usr/bin/foreman-maintain:23:in `load'
from /usr/bin/foreman-maintain:23:in `<main>'
~~~
In a similar way, backup can be interrupted when the destination filesystem runs out of space.
Actual results:
After the backup fails:
-(1)- Maintenance mode is left enabled:
~~~
# foreman-maintain maintenance-mode status
Running Status of maintenance-mode
================================================================================
Check of maintenance-mode consistency:
/ Running status of maintenance-mode
Status of maintenance-mode: On
- Iptables chain: present
- sync plans: empty data
- cron jobs: not managed [OK]
--------------------------------------------------------------------------------
~~~
-(2)- Satellite services are all down.
Expected results:
When the offline backup fails, the Satellite services are restarted and the maintenance mode is disabled.
Description of problem: When offline backup of Satellite server fails for any reason, it leaves the Satellite services in stopped status, and the maintenance mode enabled, which makes Satellite services unavailable till the following actions are manually performed: 1. Start Satellite services: ~~~ # foreman-maintain service start ~~~ 2. Disable maintenance mode: ~~~ # foreman-maintain maintenance-mode stop ~~~ Would be more efficient, and less impacting to a customer running into this situation, to have the above 2 actions done by the backup feature when it detects a failure. Version-Release number of selected component (if applicable): satellite-maintain-0.0.1-1.el7sat.noarch How reproducible: All the time reproduceable. Steps to Reproduce: 1. Launch an offline backup of Satellite server: ~~~ # foreman-maintain backup offline --assumeyes /backups ~~~ 2. Press CTRL + C in the middle of the backup operation to interrupt it: ~~~ # foreman-maintain backup offline --assumeyes /backups Starting backup: 2021-03-04 09:07:47 +0000 Running preparation steps required to run the next scenarios ================================================================================ Make sure Foreman DB is up: / Checking connection to the Foreman DB [OK] -------------------------------------------------------------------------------- Running Backup ================================================================================ Confirm turning off services is allowed: WARNING: This script will stop your services. Do you want to proceed? (assuming yes) [OK] -------------------------------------------------------------------------------- Prepare backup Directory: Creating backup folder /backups/satellite-backup-2021-03-04-09-07-47 [OK] -------------------------------------------------------------------------------- Check if the directory exists and is writable: [OK] -------------------------------------------------------------------------------- Generate metadata: / Saving metadata to metadata.yml [OK] -------------------------------------------------------------------------------- Detect features available in the local proxy: [OK] -------------------------------------------------------------------------------- disable active sync plans: \ Total 0 sync plans are now disabled. [OK] -------------------------------------------------------------------------------- Add maintenance_mode chain to iptables: [OK] -------------------------------------------------------------------------------- Stop applicable services: Stopping the following service(s): rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, rh-redis5-redis, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflow-sidekiq@orchestrator, httpd, puppetserver, dynflow-sidekiq@worker, dynflow-sidekiq@worker-hosts-queue, foreman-proxy - All services stopped [OK] -------------------------------------------------------------------------------- Backup config files: | Collecting config files to backup [OK] -------------------------------------------------------------------------------- Backup Pulp data: / Collecting Pulp data^C [RUNNING] -------------------------------------------------------------------------------- /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/utils/command_runner.rb:88:in `read': Interrupt from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/utils/command_runner.rb:88:in `block in run_non_interactively' from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/utils/command_runner.rb:83:in `popen' from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/utils/command_runner.rb:83:in `run_non_interactively' from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/utils/command_runner.rb:30:in `run' from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/concerns/system_helpers.rb:59:in `execute_runner' from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/concerns/system_helpers.rb:64:in `execute!' from /usr/share/gems/gems/foreman_maintain-0.6.15/definitions/features/tar.rb:26:in `run' from /usr/share/gems/gems/foreman_maintain-0.6.15/definitions/procedures/backup/pulp.rb:30:in `pulp_backup' from /usr/share/gems/gems/foreman_maintain-0.6.15/definitions/procedures/backup/pulp.rb:21:in `block (2 levels) in run' from /usr/share/ruby/fileutils.rb:125:in `chdir' from /usr/share/ruby/fileutils.rb:125:in `cd' from /usr/share/gems/gems/foreman_maintain-0.6.15/definitions/procedures/backup/pulp.rb:17:in `block in run' from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/reporter/cli_reporter.rb:141:in `with_spinner' from /usr/share/gems/gems/foreman_maintain-0.6.15/definitions/procedures/backup/pulp.rb:16:in `run' from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/executable.rb:139:in `__run__' from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/runner/execution.rb:83:in `block (2 levels) in run' from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/runner/execution.rb:107:in `capture_errors' from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/runner/execution.rb:82:in `block in run' from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/runner/execution.rb:101:in `with_metadata_calculation' from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/runner/execution.rb:81:in `run' from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/runner.rb:123:in `run_step' from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/runner.rb:112:in `run_steps' from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/runner.rb:105:in `execute_scenario_steps' from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/runner.rb:51:in `run_scenario' from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/runner.rb:33:in `block in run' from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/runner.rb:32:in `each' from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/runner.rb:32:in `run' from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/cli/base.rb:64:in `run_scenario' from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/cli/backup_command.rb:25:in `perform_backup' from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/cli/backup_command.rb:125:in `execute' from /usr/share/gems/gems/clamp-0.6.2/lib/clamp/command.rb:67:in `run' from /usr/share/gems/gems/clamp-0.6.2/lib/clamp/subcommand/execution.rb:11:in `execute' from /usr/share/gems/gems/clamp-0.6.2/lib/clamp/command.rb:67:in `run' from /usr/share/gems/gems/clamp-0.6.2/lib/clamp/subcommand/execution.rb:11:in `execute' from /usr/share/gems/gems/clamp-0.6.2/lib/clamp/command.rb:67:in `run' from /usr/share/gems/gems/foreman_maintain-0.6.15/lib/foreman_maintain/cli.rb:34:in `run' from /usr/share/gems/gems/clamp-0.6.2/lib/clamp/command.rb:125:in `run' from /usr/share/gems/gems/foreman_maintain-0.6.15/bin/foreman-maintain:15:in `<top (required)>' from /usr/bin/foreman-maintain:23:in `load' from /usr/bin/foreman-maintain:23:in `<main>' ~~~ In a similar way, backup can be interrupted when the destination filesystem runs out of space. Actual results: After the backup fails: -(1)- Maintenance mode is left enabled: ~~~ # foreman-maintain maintenance-mode status Running Status of maintenance-mode ================================================================================ Check of maintenance-mode consistency: / Running status of maintenance-mode Status of maintenance-mode: On - Iptables chain: present - sync plans: empty data - cron jobs: not managed [OK] -------------------------------------------------------------------------------- ~~~ -(2)- Satellite services are all down. Expected results: When the offline backup fails, the Satellite services are restarted and the maintenance mode is disabled.