Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
Satellite backup puts Satellite into maintenance mode. If the satellite-backup fails, Satellite is not taken out of maintenance mode. This causes issues when backups are taken over the weekend, and issue is not noticed until next business day.
Here is the output of the foreman backup command failing:
~~~
Starting backup: 2020-12-13 01:00:12 -0600
Running preparation steps required to run the next scenarios
================================================================================
Make sure Foreman DB is up:
/ Checking connection to the Foreman DB [OK]
--------------------------------------------------------------------------------
Running Backup
================================================================================
Confirm turning off services is allowed:
WARNING: This script will stop your services.
Do you want to proceed? (assuming yes)
[OK]
--------------------------------------------------------------------------------
Prepare backup Directory:
Creating backup folder /opt/app/backup/satellite-backup-2020-12-13-01-00-12
[OK]
--------------------------------------------------------------------------------
Check if the directory exists and is writable: /etc/profile.d/tmout.sh: line 2: TMOUT: readonly variable
[OK]
--------------------------------------------------------------------------------
Check if backup is on different logical volume then the source: [OK]
--------------------------------------------------------------------------------
Generate metadata:
| Saving metadata to metadata.yml [OK]
--------------------------------------------------------------------------------
Detect features available in the local proxy: [OK]
--------------------------------------------------------------------------------
Prepare mount point for the snapshot: [OK]
--------------------------------------------------------------------------------
disable active sync plans:
\ Total 1 sync plans are now disabled. [OK]
--------------------------------------------------------------------------------
Add maintenance_mode chain to iptables: [OK]
--------------------------------------------------------------------------------
Stop applicable services:
Stopping the following service(s):
rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflowd, httpd, puppetserver, foreman-proxy
| All services stopped [OK]
--------------------------------------------------------------------------------
Backup config files:
\ Collecting config files to backup [OK]
--------------------------------------------------------------------------------
Create and mount snapshot of Mongo DB:
\ Mounting snapshot of Mongo DB on /var/snap/mongodb [OK]
--------------------------------------------------------------------------------
Create and mount snapshot of Pulp data:
/ Creating snapshot of Pulp [FAIL]
Failed executing lvcreate -npulp-snap -L2G -s /dev/mapper/app_vg-pulp_lv, exit status 5:
Logical Volume "pulp-snap" already exists in volume group "app_vg"
--------------------------------------------------------------------------------
Scenario [Backup] failed.
The following steps ended up in failing state:
[backup-snapshot-mount-pulp]
Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="backup-snapshot-mount-pulp"
~~~
When viewing the Satellite we see iptables rule is still in place:
~~~
# iptables --list
Chain INPUT (policy ACCEPT)
target prot opt source destination
FOREMAN_MAINTAIN all -- anywhere anywhere
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain FOREMAN_MAINTAIN (1 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere
REJECT tcp -- anywhere anywhere tcp dpt:https reject-with icmp-port-unreachable
~~~
And Satellite is still in maintenance mode:
~~~
# foreman-maintain maintenance-mode status
Running Status of maintenance-mode
================================================================================
Check of maintenance-mode consistency:
/ Running status of maintenance-mode
Status of maintenance-mode: On
- Iptables chain: present
- sync plans: disabled
- cron jobs: not managed [OK]
--------------------------------------------------------------------------------
~~~
Version-Release number of selected component (if applicable):
Satellite 6.8
How reproducible:
always when backup fails due to (Logical Volume "pulp-snap" already exists in volume group "app_vg")
Steps to Reproduce:
1. have satellite backup fail due to (Logical Volume "pulp-snap" already exists in volume group "app_vg")
2. check status of Satellite
3.
Actual results:
Satellite is still in maintenance mode
Expected results:
Expect the Satellite to try to go back to operational if back up fails.
Additional info:
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (Satellite Maintenance 6.9.3 Async Bug Fix Update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2021:2635
Description of problem: Satellite backup puts Satellite into maintenance mode. If the satellite-backup fails, Satellite is not taken out of maintenance mode. This causes issues when backups are taken over the weekend, and issue is not noticed until next business day. Here is the output of the foreman backup command failing: ~~~ Starting backup: 2020-12-13 01:00:12 -0600 Running preparation steps required to run the next scenarios ================================================================================ Make sure Foreman DB is up: / Checking connection to the Foreman DB [OK] -------------------------------------------------------------------------------- Running Backup ================================================================================ Confirm turning off services is allowed: WARNING: This script will stop your services. Do you want to proceed? (assuming yes) [OK] -------------------------------------------------------------------------------- Prepare backup Directory: Creating backup folder /opt/app/backup/satellite-backup-2020-12-13-01-00-12 [OK] -------------------------------------------------------------------------------- Check if the directory exists and is writable: /etc/profile.d/tmout.sh: line 2: TMOUT: readonly variable [OK] -------------------------------------------------------------------------------- Check if backup is on different logical volume then the source: [OK] -------------------------------------------------------------------------------- Generate metadata: | Saving metadata to metadata.yml [OK] -------------------------------------------------------------------------------- Detect features available in the local proxy: [OK] -------------------------------------------------------------------------------- Prepare mount point for the snapshot: [OK] -------------------------------------------------------------------------------- disable active sync plans: \ Total 1 sync plans are now disabled. [OK] -------------------------------------------------------------------------------- Add maintenance_mode chain to iptables: [OK] -------------------------------------------------------------------------------- Stop applicable services: Stopping the following service(s): rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflowd, httpd, puppetserver, foreman-proxy | All services stopped [OK] -------------------------------------------------------------------------------- Backup config files: \ Collecting config files to backup [OK] -------------------------------------------------------------------------------- Create and mount snapshot of Mongo DB: \ Mounting snapshot of Mongo DB on /var/snap/mongodb [OK] -------------------------------------------------------------------------------- Create and mount snapshot of Pulp data: / Creating snapshot of Pulp [FAIL] Failed executing lvcreate -npulp-snap -L2G -s /dev/mapper/app_vg-pulp_lv, exit status 5: Logical Volume "pulp-snap" already exists in volume group "app_vg" -------------------------------------------------------------------------------- Scenario [Backup] failed. The following steps ended up in failing state: [backup-snapshot-mount-pulp] Resolve the failed steps and rerun the command. In case the failures are false positives, use --whitelist="backup-snapshot-mount-pulp" ~~~ When viewing the Satellite we see iptables rule is still in place: ~~~ # iptables --list Chain INPUT (policy ACCEPT) target prot opt source destination FOREMAN_MAINTAIN all -- anywhere anywhere Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain FOREMAN_MAINTAIN (1 references) target prot opt source destination ACCEPT all -- anywhere anywhere REJECT tcp -- anywhere anywhere tcp dpt:https reject-with icmp-port-unreachable ~~~ And Satellite is still in maintenance mode: ~~~ # foreman-maintain maintenance-mode status Running Status of maintenance-mode ================================================================================ Check of maintenance-mode consistency: / Running status of maintenance-mode Status of maintenance-mode: On - Iptables chain: present - sync plans: disabled - cron jobs: not managed [OK] -------------------------------------------------------------------------------- ~~~ Version-Release number of selected component (if applicable): Satellite 6.8 How reproducible: always when backup fails due to (Logical Volume "pulp-snap" already exists in volume group "app_vg") Steps to Reproduce: 1. have satellite backup fail due to (Logical Volume "pulp-snap" already exists in volume group "app_vg") 2. check status of Satellite 3. Actual results: Satellite is still in maintenance mode Expected results: Expect the Satellite to try to go back to operational if back up fails. Additional info: