Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1775665

Summary: [RFE] ipa-backup might leave a system in offline state when the backup failed
Product: Red Hat Enterprise Linux 8 Reporter: Thorsten Scherf <tscherf>
Component: ipaAssignee: Thomas Woerner <twoerner>
Status: CLOSED WONTFIX QA Contact: ipa-qe <ipa-qe>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 8.1CC: fcami, rcritten, tscherf
Target Milestone: rcKeywords: FutureFeature
Target Release: 8.0Flags: pm-rhel: mirror+
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-01-30 18:01:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1796589    
Bug Blocks:    

Description Thorsten Scherf 2019-11-22 14:30:53 UTC
Description of problem:
During the backup process 'ipa-backup' tool uses the /tmp filesystem to store some temporary data. When there is insufficient space on the filesystem, the backup fails and the tool terminates without restarting the IdM service again.

2019-11-08T05:03:40Z DEBUG Process finished, return code=2
2019-11-08T05:03:40Z DEBUG stdout=
2019-11-08T05:03:40Z DEBUG stderr=tar: Removing leading `/' from member names
tar: Removing leading `/' from hard link targets
tar: /tmp/tmp114rVbipa/ipa/files.tar: Cannot write: No space left on device
tar: Error is not recoverable: exiting now


Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1.
2.
3.

Actual results:
The backup tool fails and the IdM service is offline.

Expected results:
In case of a failure, the tool should start the IdM service again and print a warning that the backup failed. 

Additional info:
We have some advise in the product docs how to workaround the problem of insufficient disk space for the backup process:

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/linux_domain_identity_authentication_and_policy_guide/index#create-full-backup

But a backup tool should never leave the actual service in offline state even when the actual backup failed.

Comment 1 François Cami 2019-12-02 10:06:21 UTC
I think the current behavior is fine because:
* not restarting services (e.g. having services down) should alert an IPA administrator that something is wrong (ideally via monitoring)
* restarting services (esp. DS) without sufficient disk space might be problematic (ENOSPC possibly leading to DB corruption).
The behavior is of course not ideal but at least an administrator would have to take action and understand why the backups have been failing.

There is a lot of room for improvement:
* it should be doable to compute how much space temporary files for a backup would take, as we know how much each entry would consume
* this could be used in turn by both ipa-backup and ipa-healthcheck.

Thorsten, do you agree with the above?

Comment 2 Thorsten Scherf 2019-12-02 10:16:56 UTC
(In reply to François Cami from comment #1)
> I think the current behavior is fine because:
> * not restarting services (e.g. having services down) should alert an IPA
> administrator that something is wrong (ideally via monitoring)

A failed backup should not lead to a complete service down situation.

> * restarting services (esp. DS) without sufficient disk space might be
> problematic (ENOSPC possibly leading to DB corruption).

We are talking about temporary disk space that is required to hold temporary files while the backup is created. These files are usually stored in /tmp (unless TMPDIR is set to some other folder). 

> The behavior is of course not ideal but at least an administrator would have
> to take action and understand why the backups have been failing.
> 
> There is a lot of room for improvement:
> * it should be doable to compute how much space temporary files for a backup
> would take, as we know how much each entry would consume

Yes, we should calculate how much space is needed for the temporary files used by the backup tool. When less space is available the tool should give a warning with the options to a) either free disk space in /tmp or b) set TMPDIR variable to a different directory that has sufficient disk space (only for the time the backup is running). Then the tool should exit.
  
> * this could be used in turn by both ipa-backup and ipa-healthcheck.
> 
> Thorsten, do you agree with the above?

See comments above.

Comment 3 François Cami 2019-12-02 10:21:07 UTC
> These files are usually stored in /tmp 

Agreed, but there is no check either that /tmp and /var/lib are separate file systems. This is why I argue it is best to leave services off at the time.

> Yes, we should calculate how much space is needed for the temporary files used by the backup tool.

I think this is where we'll be going as a general enhancement. It will solve the above issue.

Comment 5 François Cami 2020-01-30 18:01:59 UTC
This was discussed in the FreeIPA triage meeting this week and the consensus is that if ipa-backup fails, IPA services should be left in the stopped state due to the concerns listed in  https://bugzilla.redhat.com/show_bug.cgi?id=1775665#c3 . Therefore I am closing this bug as WONTFIX.

I have however opened https://bugzilla.redhat.com/show_bug.cgi?id=1796589 to avoid such situations.