Red Hat Bugzilla – Bug 1445631
[downstream clone - 4.1.2] [backup] Engine backup stages to /tmp, can fill disk to 100% and down rhv-m
Last modified: 2017-07-06 03:30:42 EDT
+++ This bug is a downstream clone. The original bug is: +++ +++ bug 1439332 +++ ====================================================================== (Filing on behalf of customer) Summary: Engine backup currently stages data to /tmp, which in an emergency backup situation (such as a filling disk to due DWH overflow) is a problem, because it can fill the disk up and crash the manager. Description: $customer had a situation where the filesystem on HostedEngine was near 99% full with only 1.2GB left of space. They wanted to create an engine backup with scope=all and make the destination of the backup to an NFS share (since they didn't have space on my engine). The command they used for engine-backup is below (note that /engine-backup is an nfs mounted directory) /usr/bin/engine-backup --mode=backup --scope=all --archive-compressor=gzip --file=/engine-backups/dbdump/EngineDBbkp-$(hostname)-$(date '+%Y%m%d.%H%M').gzip --log=/var/tmp/EngineDBbkp-$(date '+%Y%m%d.%H%M').log Customer noticed it was writing to /tmp filesystem (through the backup log), which led it to filling up HostedEngine's filesystem to 100%. The engine-backup should not stage files within /tmp, it should instead only utilize the location specified in the '--file=' parameter passed to engine-backup. If a staging directory is needed, perhaps add an option to specify the staging directory as well? You can see in the log below from the engine-backup that the target was /engine-backup directory but later in the log you see it staging all the files in a temporary directory in /tmp/engine-backup.vhDTVpcYlG/ 2017-04-04 08:31:59 1072: Start of engine-backup mode backup scope all file /engine-backups/dbdump/EngineDBbkp-rhvm-engine01.[redacted]-20170404.0831.gzip 2017-04-04 08:31:59 1072: OUTPUT: Backing up: 2017-04-04 08:31:59 1072: Generating pgpass 2017-04-04 08:31:59 1072: OUTPUT: Notifying engine 2017-04-04 08:31:59 1072: pg_cmd running: psql -w -U engine -h localhost -p 5432 engine -t -c SELECT LogEngineBackupEvent('files', now(), 0, 'Started', 'rhvm-engine01.[redacted]', '/var/tmp/EngineDBbkp-20170404.0831.log'); 2017-04-04 08:31:59 1072: pg_cmd running: psql -w -U engine -h localhost -p 5432 engine -t -c SELECT LogEngineBackupEvent('db', now(), 0, 'Started', 'rhvm-engine01.[redacted]', '/var/tmp/EngineDBbkp-20170404.0831.log'); 2017-04-04 08:31:59 1072: pg_cmd running: psql -w -U engine -h localhost -p 5432 engine -t -c SELECT LogEngineBackupEvent('dwhdb', now(), 0, 'Started', 'rhvm-engine01.[redacted]', '/var/tmp/EngineDBbkp-20170404.0831.log'); 2017-04-04 08:31:59 1072: Creating temp folder /tmp/engine-backup.vhDTVpcYlG/tar 2017-04-04 08:31:59 1072: OUTPUT: - Files 2017-04-04 08:31:59 1072: Backing up files to /tmp/engine-backup.vhDTVpcYlG/tar/files 2017-04-04 08:32:01 1072: OUTPUT: - Engine database 'engine' 2017-04-04 08:32:01 1072: Backing up database to /tmp/engine-backup.vhDTVpcYlG/tar/db/engine_backup.db 2017-04-04 08:32:01 1072: pg_cmd running: pg_dump -w -U engine -h localhost -p 5432 engine -E UTF8 --disable-dollar-quoting --disable-triggers --format=custom 2017-04-04 08:32:05 1072: OUTPUT: - DWH database 'ovirt_engine_history' 2017-04-04 08:32:05 1072: Backing up dwh database to /tmp/engine-backup.vhDTVpcYlG/tar/db/dwh_backup.db 2017-04-04 08:32:05 1072: pg_cmd running: pg_dump -w -U ovirt_engine_history -h localhost -p 5432 ovirt_engine_history -E UTF8 --disable-dollar-quoting --disable-triggers --format=custom Where are you experiencing the behavior? What environment? RHVM 4.0.6 Hypervisors: RHEL 7.3 (RHEV 4.0) Reprodicibility: every time (Originally by Ashton Davis)
Changing temp dir is already possible. engine-backup honor system temporary folder defined in $TMPDIR. If a different tmpdir is needed, a TMPDIR=/path/to/tmpdir engine-backup <args> should be enough. Is this enough or do we still need to consider to set TMPDIR to dirname of --file argument? (Originally by Sandro Bonazzola)
The feedback I'm getting is that because it's not obvious that tmp dir is configurable, it would be better to have it default to staging at the target directory, with an option to stage in local /tmp. (Originally by Ashton Davis)
(In reply to Sandro Bonazzola from comment #1) > Changing temp dir is already possible. engine-backup honor system temporary > folder defined in $TMPDIR. Actually it does not, due to the fix for bug 1172191. If you want to workaround this, and know that your environment is otherwise clean, you can do something like that: BACKUP_ENV_CLEAN=1 TMPDIR=/var/tmp engine-backup --mode=backup --file=f1 --log=l1 (In reply to Ashton Davis from comment #2) > The feedback I'm getting is that because it's not obvious that tmp dir is > configurable, This can also be handled as a documentation bug, if it worked. > it would be better to have it default to staging at the target > directory, with an option to stage in local /tmp. Please note that there is no "target directory". Only a "target file". These are in principle very different. E.g., the following: cd /root ln -s /tmp/f1 f1 engine-backup --file=f1 --log=l1 --mode=backup will keep /root/f1 a symlink, and the actual archive will thus be written to /tmp/f1. So in such a case, which is the "target directory"? "/root", or "/tmp"? And should it be engine-backup that decides about this (perhaps based on our current discussion), or we need yet another option ("--follow-symlinks-when-deciding-about-tmpdir") to allow the user to choose? (Originally by didi)
How about simply having a new option --tmpdir=/some/temp/dir, which will default to $TMPDIR if set, or to /tmp otherwise? To make it clear - it will behave like /tmp. That is, it must exist, and be a directory, before engine-backup starts, and engine-backup will not delete it. This is to prevent cases like bug 1288197 and bug 1199147. (Originally by didi)
(In reply to Yedidyah Bar David from comment #4) > How about simply having a new option --tmpdir=/some/temp/dir, which will > default to $TMPDIR if set, or to /tmp otherwise? > Seems like the most straight forward option. (Originally by Yaniv Dary)
*** Bug 1415623 has been marked as a duplicate of this bug. *** (Originally by Sandro Bonazzola)
(In reply to Yaniv Dary from comment #5) > (In reply to Yedidyah Bar David from comment #4) > > How about simply having a new option --tmpdir=/some/temp/dir, which will > > default to $TMPDIR if set, or to /tmp otherwise? > > > > Seems like the most straight forward option. I concur, this makes the most sense. It seems to both get around the target file -vs- dir issue, and still gives the user choice for where to stage the transfer. (Originally by Ashton Davis)
Verified on ovirt-engine-4.1.3.2-0.1.el7.noarch # engine-backup --help | grep tmpdir --tmpdir=DIR Set temporary directory parent. See also mktemp(1). # mkdir /var/backup_tmpdir # engine-backup --tmpdir=/var/backup_tmpdir --mode=backup --file=backup --log=backup.log Backing up: Notifying engine - Files - Engine database 'engine' - DWH database 'ovirt_engine_history' Packing into file 'backig' Notifying engine Done.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1692