Bug 1693608

Summary: Rear daily cron run fills temp directory upon failure
Product: Red Hat Enterprise Linux 7 Reporter: Oliver Falk <ofalk>
Component: rearAssignee: Pavel Cahyna <pcahyna>
Status: CLOSED ERRATA QA Contact: David Jež <djez>
Severity: high Docs Contact: Mariya Pershina <mpershin>
Priority: unspecified    
Version: 7.6CC: djez, lkuprova, mdolezel, ovasik, pcahyna, peter.vreman
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: rear-2.4-11.el7 Doc Type: Bug Fix
Doc Text:
.`ReaR` updates RHEL 7.8 introduces a number of updates to the Relax-and-Recover (`ReaR`) utility. The build directory handling has been changed. Previously, the build directory was kept in a temporary location in case `ReaR` encountered a failure. With this update, the build directory is deleted by default in non-interactive runs to prevent consuming disk space. The semantics of the `KEEP_BUILD_DIR` configuration variable has been enhanced to include a new `errors` value. You can set the `KEEP_BUILD_DIR` variable to the following values: * `errors` to preserve the build directory on errors for debugging (the previous behavior) * `y` (`true`) to always preserve the build directory * `n` (`false`) to never preserve the build directory The default value is an empty string with the meaning of `errors` when `ReaR` is being executed interactively (in a terminal) and `false` if `ReaR` is being executed non-interactively. Note that `KEEP_BUILD_DIR` is automatically set to `true` in debug mode (`-d`) and in debugscript mode (`-D`); this behavior has not been changed. Notable bug fixes include: * Support for NetBackup 8.0 has been fixed. * `ReaR` no longer aborts with a bash error similar to `xrealloc: cannot allocate` on systems with a large number of users, groups, and users per group. * The `bconsole` command now shows its prompt, which enables you to perform a restore operation when using the Bacula integration. * `ReaR` now correctly backs up files also in situations when the `docker` service is running but no `docker` root directory has been defined, or when it is impossible to determine the status of the `docker` service. * Recovery no longer fails when using thin pools or recovering a system in Migration Mode. * Extremely slow rebuild of `initramfs` during the recovery process with LVM has been fixed.
Story Points: ---
Clone Of:
: 1729501 (view as bug list) Environment:
Last Closed: 2020-03-31 20:06:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1122832, 1716965, 1729501    

Description Oliver Falk 2019-03-28 10:02:58 UTC
Description of problem:
If rear fails, for whatever reasons, it leaves behind the temporary files/directories. This can easily fill up the disk space and lead to service interruptions => Therefore severity high.


Version-Release number of selected component (if applicable): 2.4


How reproducible: Always


Steps to Reproduce:
1. Install rear
2. Make sure rear fails (eg. make sure some binary is included that uses a library, that is not automatically included)
3. Run sudo rear mkrescue -v


Actual results:
Running mkrescue -v gives you the right hint already:

"""
You should also rm -Rf /var/tmp/rear.Vow2ZOUr9jUanAG
"""

While this is perfectly fine when running rear interactively, running it via cron should always clean up the temporary files.

Expected results:
Rear cron job shouldn't leave behind temporary directories, when not run interactively.


Additional info:
A probable workaround could be to source the rear config and remove $TMPDIR/rear.*. Eventually check if $TMPDIR is set before; Else TMPDIR => /tmp.

Customer case will be attached in a hurry.

Comment 3 Pavel Cahyna 2019-08-27 12:57:43 UTC
Some notes:
- this behavior is intentional, ReaR leaves around the build directory for debugging in case of errors. See the upstream issue https://github.com/rear/rear/issues/2121 where a user complained about it and the "solution" was just to inform about it in a comment in the default configuration file.
- it should not be the case that daily cron runs will fill the temp directory. The crontab contains
/usr/sbin/rear checklayout || /usr/sbin/rear mkrescue
so the build directory will be left on first run, but on the subsequent ones checklayout will succeed and further build directories should not be produced, so the used space should not grow without bounds and the damage should be limited (unless the disk layout changes often for some reason).
- even given the above, I agree that the current behavior is unfortunate and I submitted PR2218 upstream to not leave the build directory around in noninteractive runs and warn more prominently about it (without the -v flag) in interactive runs.

Comment 4 David Jež 2019-08-29 09:56:06 UTC
Pavle,
Let's add a new option KEEP_BUILD_DIR_ON_ERRORS. If set to yes, keep build dir on errors even in non interactive mode, which is really useful for troubleshoot. If empty, fill KEEP_BUILD_DIR based on tty. Only KEEP_BUILD_DIR itself could not be enough because it always remove directory or always keep it which is not so useful for troubleshoot.

Thank you,
DJ

Comment 5 Pavel Cahyna 2019-09-02 15:42:22 UTC
Instead of introducing a new variable KEEP_BUILD_DIR_ON_ERRORS, we plan with upstream to introduce a new possible value for KEEP_BUILD_DIR, KEEP_BUILD_DIR="errors", which will have the same semantics as the proposed KEEP_BUILD_DIR_ON_ERRORS=yes. See https://github.com/rear/rear/pull/2218#issuecomment-526679705

Comment 13 Pavel Cahyna 2019-09-13 12:44:04 UTC
PR merged upstream without modification.

Comment 22 errata-xmlrpc 2020-03-31 20:06:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1153