Bug 1439332 - [backup] Engine backup stages to /tmp, can fill disk to 100% and down rhv-m
Summary: [backup] Engine backup stages to /tmp, can fill disk to 100% and down rhv-m
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 4.0.6
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ovirt-4.2.0
: ---
Assignee: Yedidyah Bar David
QA Contact: Lukas Svaty
URL:
Whiteboard:
: 1415623 (view as bug list)
Depends On:
Blocks: 1445631
TreeView+ depends on / blocked
 
Reported: 2017-04-05 17:54 UTC by Ashton Davis
Modified: 2020-05-14 15:53 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Previously, engine-backup always used the /tmp directory, which was problematic if the directory was full. In the current release, it is possible to change the engine-backup's temporary directory.
Clone Of:
: 1445631 (view as bug list)
Environment:
Last Closed: 2018-05-15 17:41:54 UTC
oVirt Team: Integration
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1415623 0 medium CLOSED [RFE] posiblity to change temp folder of engine-backup 2021-02-22 00:41:40 UTC
Red Hat Product Errata RHEA-2018:1488 0 None None None 2018-05-15 17:43:01 UTC
oVirt gerrit 75589 0 'None' MERGED packaging: engine-backup: Allow setting temp dir 2020-08-06 00:51:46 UTC

Internal Links: 1415623

Description Ashton Davis 2017-04-05 17:54:09 UTC
(Filing on behalf of customer)

Summary: Engine backup currently stages data to /tmp, which in an emergency backup situation (such as a filling disk to due DWH overflow) is a problem, because it can fill the disk up and crash the manager.

Description:

$customer had a situation where the filesystem on HostedEngine was near 99% full with only 1.2GB left of space. They wanted to create an engine backup with scope=all and make the destination of the backup to an NFS share (since they didn't have space on my engine). The command they used for engine-backup is below (note that /engine-backup is an nfs mounted directory)

/usr/bin/engine-backup --mode=backup --scope=all --archive-compressor=gzip --file=/engine-backups/dbdump/EngineDBbkp-$(hostname)-$(date '+%Y%m%d.%H%M').gzip --log=/var/tmp/EngineDBbkp-$(date '+%Y%m%d.%H%M').log


Customer noticed it was writing to /tmp filesystem (through the backup log), which led it to filling up HostedEngine's filesystem to 100%.

The engine-backup should not stage files within /tmp, it should instead only utilize the location specified in the '--file=' parameter passed to engine-backup. If a staging directory is needed, perhaps add an option to specify the staging directory as well? 

You can see in the log below from the engine-backup that the target was /engine-backup directory but later in the log you see it staging all the files in a temporary directory in  /tmp/engine-backup.vhDTVpcYlG/



2017-04-04 08:31:59 1072: Start of engine-backup mode backup scope all file /engine-backups/dbdump/EngineDBbkp-rhvm-engine01.[redacted]-20170404.0831.gzip
2017-04-04 08:31:59 1072: OUTPUT: Backing up:
2017-04-04 08:31:59 1072: Generating pgpass
2017-04-04 08:31:59 1072: OUTPUT: Notifying engine
2017-04-04 08:31:59 1072: pg_cmd running: psql -w -U engine -h localhost -p 5432  engine -t -c SELECT LogEngineBackupEvent('files', now(), 0, 'Started', 'rhvm-engine01.[redacted]', '/var/tmp/EngineDBbkp-20170404.0831.log');


2017-04-04 08:31:59 1072: pg_cmd running: psql -w -U engine -h localhost -p 5432  engine -t -c SELECT LogEngineBackupEvent('db', now(), 0, 'Started', 'rhvm-engine01.[redacted]', '/var/tmp/EngineDBbkp-20170404.0831.log');


2017-04-04 08:31:59 1072: pg_cmd running: psql -w -U engine -h localhost -p 5432  engine -t -c SELECT LogEngineBackupEvent('dwhdb', now(), 0, 'Started', 'rhvm-engine01.[redacted]', '/var/tmp/EngineDBbkp-20170404.0831.log');


2017-04-04 08:31:59 1072: Creating temp folder /tmp/engine-backup.vhDTVpcYlG/tar
2017-04-04 08:31:59 1072: OUTPUT: - Files
2017-04-04 08:31:59 1072: Backing up files to /tmp/engine-backup.vhDTVpcYlG/tar/files
2017-04-04 08:32:01 1072: OUTPUT: - Engine database 'engine'
2017-04-04 08:32:01 1072: Backing up database to /tmp/engine-backup.vhDTVpcYlG/tar/db/engine_backup.db
2017-04-04 08:32:01 1072: pg_cmd running: pg_dump -w -U engine -h localhost -p 5432  engine -E UTF8 --disable-dollar-quoting --disable-triggers --format=custom
2017-04-04 08:32:05 1072: OUTPUT: - DWH database 'ovirt_engine_history'
2017-04-04 08:32:05 1072: Backing up dwh database to /tmp/engine-backup.vhDTVpcYlG/tar/db/dwh_backup.db
2017-04-04 08:32:05 1072: pg_cmd running: pg_dump -w -U ovirt_engine_history -h localhost -p 5432  ovirt_engine_history -E UTF8 --disable-dollar-quoting --disable-triggers --format=custom

Where are you experiencing the behavior?  What environment?

RHVM 4.0.6
Hypervisors: RHEL 7.3 (RHEV 4.0)

Reprodicibility: 
every time

Comment 1 Sandro Bonazzola 2017-04-10 07:18:29 UTC
Changing temp dir is already possible. engine-backup honor system temporary folder defined in $TMPDIR.
If a different tmpdir is needed, a TMPDIR=/path/to/tmpdir engine-backup <args> should be enough.

Is this enough or do we still need to consider to set TMPDIR to dirname of --file argument?

Comment 2 Ashton Davis 2017-04-11 14:34:34 UTC
The feedback I'm getting is that because it's not obvious that tmp dir is configurable, it would be better to have it default to staging at the target directory, with an option to stage in local /tmp.

Comment 3 Yedidyah Bar David 2017-04-12 06:24:24 UTC
(In reply to Sandro Bonazzola from comment #1)
> Changing temp dir is already possible. engine-backup honor system temporary
> folder defined in $TMPDIR.

Actually it does not, due to the fix for bug 1172191.

If you want to workaround this, and know that your environment is otherwise clean, you can do something like that:

BACKUP_ENV_CLEAN=1 TMPDIR=/var/tmp engine-backup --mode=backup --file=f1 --log=l1

(In reply to Ashton Davis from comment #2)
> The feedback I'm getting is that because it's not obvious that tmp dir is
> configurable,

This can also be handled as a documentation bug, if it worked.

> it would be better to have it default to staging at the target
> directory, with an option to stage in local /tmp.

Please note that there is no "target directory". Only a "target file". These are in principle very different. E.g., the following:

cd /root
ln -s /tmp/f1 f1
engine-backup --file=f1 --log=l1 --mode=backup

will keep /root/f1 a symlink, and the actual archive will thus be written to /tmp/f1. So in such a case, which is the "target directory"? "/root", or "/tmp"? And should it be engine-backup that decides about this (perhaps based on our current discussion), or we need yet another option ("--follow-symlinks-when-deciding-about-tmpdir") to allow the user to choose?

Comment 4 Yedidyah Bar David 2017-04-12 06:24:55 UTC
How about simply having a new option --tmpdir=/some/temp/dir, which will default to $TMPDIR if set, or to /tmp otherwise?

To make it clear - it will behave like /tmp. That is, it must exist, and be a directory, before engine-backup starts, and engine-backup will not delete it. This is to prevent cases like bug 1288197 and bug 1199147.

Comment 5 Yaniv Lavi 2017-04-12 11:04:32 UTC
(In reply to Yedidyah Bar David from comment #4)
> How about simply having a new option --tmpdir=/some/temp/dir, which will
> default to $TMPDIR if set, or to /tmp otherwise?
> 

Seems like the most straight forward option.

Comment 6 Sandro Bonazzola 2017-04-13 07:30:42 UTC
*** Bug 1415623 has been marked as a duplicate of this bug. ***

Comment 7 Ashton Davis 2017-04-13 19:01:24 UTC
(In reply to Yaniv Dary from comment #5)
> (In reply to Yedidyah Bar David from comment #4)
> > How about simply having a new option --tmpdir=/some/temp/dir, which will
> > default to $TMPDIR if set, or to /tmp otherwise?
> > 
> 
> Seems like the most straight forward option.

I concur, this makes the most sense. It seems to both get around the target file -vs- dir issue, and still gives the user choice for where to stage the transfer.

Comment 10 Lukas Svaty 2018-01-26 14:58:15 UTC
--tmpdir param working and used instead of default /tmp

verified in ovirt-engine-tools-backup-4.2.1.3-0.1.el7.noarch

Comment 13 errata-xmlrpc 2018-05-15 17:41:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:1488

Comment 14 Franta Kust 2019-05-16 13:07:48 UTC
BZ<2>Jira Resync


Note You need to log in before you can comment on or make changes to this bug.