Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1445631

Summary: [downstream clone - 4.1.2] [backup] Engine backup stages to /tmp, can fill disk to 100% and down rhv-m
Product: Red Hat Enterprise Virtualization Manager Reporter: rhev-integ
Component: ovirt-engineAssignee: Yedidyah Bar David <didi>
Status: CLOSED ERRATA QA Contact: Aleksei Slaikovskii <aslaikov>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.0.6CC: asdavis, jspanko, lsurette, melewis, pstehlik, rbalakri, Rhev-m-bugs, srevivo, ykaul, ylavi
Target Milestone: ovirt-4.1.3Keywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
With this update, the user can now control the location of the temporary directory that the engine-backup command creates. The command line option --tmpdir=DIR and the variable $TMPDIR have been added to control the location of the temporary directory. Previously, the engine-backup command always used /tmp. If /tmp was full, or close to being full, it was not possible to tell the engine-backup command to use a different directory. Now, it is possible to run the engine-backup command even if the /tmp directory is full by telling the engine-backup command to use a different location.
Story Points: ---
Clone Of: 1439332 Environment:
Last Closed: 2017-07-06 07:30:42 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Integration RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1439332    
Bug Blocks:    

Description rhev-integ 2017-04-26 07:53:31 UTC
+++ This bug is a downstream clone. The original bug is: +++
+++   bug 1439332 +++
======================================================================

(Filing on behalf of customer)

Summary: Engine backup currently stages data to /tmp, which in an emergency backup situation (such as a filling disk to due DWH overflow) is a problem, because it can fill the disk up and crash the manager.

Description:

$customer had a situation where the filesystem on HostedEngine was near 99% full with only 1.2GB left of space. They wanted to create an engine backup with scope=all and make the destination of the backup to an NFS share (since they didn't have space on my engine). The command they used for engine-backup is below (note that /engine-backup is an nfs mounted directory)

/usr/bin/engine-backup --mode=backup --scope=all --archive-compressor=gzip --file=/engine-backups/dbdump/EngineDBbkp-$(hostname)-$(date '+%Y%m%d.%H%M').gzip --log=/var/tmp/EngineDBbkp-$(date '+%Y%m%d.%H%M').log


Customer noticed it was writing to /tmp filesystem (through the backup log), which led it to filling up HostedEngine's filesystem to 100%.

The engine-backup should not stage files within /tmp, it should instead only utilize the location specified in the '--file=' parameter passed to engine-backup. If a staging directory is needed, perhaps add an option to specify the staging directory as well? 

You can see in the log below from the engine-backup that the target was /engine-backup directory but later in the log you see it staging all the files in a temporary directory in  /tmp/engine-backup.vhDTVpcYlG/



2017-04-04 08:31:59 1072: Start of engine-backup mode backup scope all file /engine-backups/dbdump/EngineDBbkp-rhvm-engine01.[redacted]-20170404.0831.gzip
2017-04-04 08:31:59 1072: OUTPUT: Backing up:
2017-04-04 08:31:59 1072: Generating pgpass
2017-04-04 08:31:59 1072: OUTPUT: Notifying engine
2017-04-04 08:31:59 1072: pg_cmd running: psql -w -U engine -h localhost -p 5432  engine -t -c SELECT LogEngineBackupEvent('files', now(), 0, 'Started', 'rhvm-engine01.[redacted]', '/var/tmp/EngineDBbkp-20170404.0831.log');


2017-04-04 08:31:59 1072: pg_cmd running: psql -w -U engine -h localhost -p 5432  engine -t -c SELECT LogEngineBackupEvent('db', now(), 0, 'Started', 'rhvm-engine01.[redacted]', '/var/tmp/EngineDBbkp-20170404.0831.log');


2017-04-04 08:31:59 1072: pg_cmd running: psql -w -U engine -h localhost -p 5432  engine -t -c SELECT LogEngineBackupEvent('dwhdb', now(), 0, 'Started', 'rhvm-engine01.[redacted]', '/var/tmp/EngineDBbkp-20170404.0831.log');


2017-04-04 08:31:59 1072: Creating temp folder /tmp/engine-backup.vhDTVpcYlG/tar
2017-04-04 08:31:59 1072: OUTPUT: - Files
2017-04-04 08:31:59 1072: Backing up files to /tmp/engine-backup.vhDTVpcYlG/tar/files
2017-04-04 08:32:01 1072: OUTPUT: - Engine database 'engine'
2017-04-04 08:32:01 1072: Backing up database to /tmp/engine-backup.vhDTVpcYlG/tar/db/engine_backup.db
2017-04-04 08:32:01 1072: pg_cmd running: pg_dump -w -U engine -h localhost -p 5432  engine -E UTF8 --disable-dollar-quoting --disable-triggers --format=custom
2017-04-04 08:32:05 1072: OUTPUT: - DWH database 'ovirt_engine_history'
2017-04-04 08:32:05 1072: Backing up dwh database to /tmp/engine-backup.vhDTVpcYlG/tar/db/dwh_backup.db
2017-04-04 08:32:05 1072: pg_cmd running: pg_dump -w -U ovirt_engine_history -h localhost -p 5432  ovirt_engine_history -E UTF8 --disable-dollar-quoting --disable-triggers --format=custom

Where are you experiencing the behavior?  What environment?

RHVM 4.0.6
Hypervisors: RHEL 7.3 (RHEV 4.0)

Reprodicibility: 
every time

(Originally by Ashton Davis)

Comment 1 rhev-integ 2017-04-26 07:53:40 UTC
Changing temp dir is already possible. engine-backup honor system temporary folder defined in $TMPDIR.
If a different tmpdir is needed, a TMPDIR=/path/to/tmpdir engine-backup <args> should be enough.

Is this enough or do we still need to consider to set TMPDIR to dirname of --file argument?

(Originally by Sandro Bonazzola)

Comment 3 rhev-integ 2017-04-26 07:53:45 UTC
The feedback I'm getting is that because it's not obvious that tmp dir is configurable, it would be better to have it default to staging at the target directory, with an option to stage in local /tmp.

(Originally by Ashton Davis)

Comment 4 rhev-integ 2017-04-26 07:53:51 UTC
(In reply to Sandro Bonazzola from comment #1)
> Changing temp dir is already possible. engine-backup honor system temporary
> folder defined in $TMPDIR.

Actually it does not, due to the fix for bug 1172191.

If you want to workaround this, and know that your environment is otherwise clean, you can do something like that:

BACKUP_ENV_CLEAN=1 TMPDIR=/var/tmp engine-backup --mode=backup --file=f1 --log=l1

(In reply to Ashton Davis from comment #2)
> The feedback I'm getting is that because it's not obvious that tmp dir is
> configurable,

This can also be handled as a documentation bug, if it worked.

> it would be better to have it default to staging at the target
> directory, with an option to stage in local /tmp.

Please note that there is no "target directory". Only a "target file". These are in principle very different. E.g., the following:

cd /root
ln -s /tmp/f1 f1
engine-backup --file=f1 --log=l1 --mode=backup

will keep /root/f1 a symlink, and the actual archive will thus be written to /tmp/f1. So in such a case, which is the "target directory"? "/root", or "/tmp"? And should it be engine-backup that decides about this (perhaps based on our current discussion), or we need yet another option ("--follow-symlinks-when-deciding-about-tmpdir") to allow the user to choose?

(Originally by didi)

Comment 5 rhev-integ 2017-04-26 07:53:56 UTC
How about simply having a new option --tmpdir=/some/temp/dir, which will default to $TMPDIR if set, or to /tmp otherwise?

To make it clear - it will behave like /tmp. That is, it must exist, and be a directory, before engine-backup starts, and engine-backup will not delete it. This is to prevent cases like bug 1288197 and bug 1199147.

(Originally by didi)

Comment 6 rhev-integ 2017-04-26 07:54:01 UTC
(In reply to Yedidyah Bar David from comment #4)
> How about simply having a new option --tmpdir=/some/temp/dir, which will
> default to $TMPDIR if set, or to /tmp otherwise?
> 

Seems like the most straight forward option.

(Originally by Yaniv Dary)

Comment 7 rhev-integ 2017-04-26 07:54:05 UTC
*** Bug 1415623 has been marked as a duplicate of this bug. ***

(Originally by Sandro Bonazzola)

Comment 8 rhev-integ 2017-04-26 07:54:11 UTC
(In reply to Yaniv Dary from comment #5)
> (In reply to Yedidyah Bar David from comment #4)
> > How about simply having a new option --tmpdir=/some/temp/dir, which will
> > default to $TMPDIR if set, or to /tmp otherwise?
> > 
> 
> Seems like the most straight forward option.

I concur, this makes the most sense. It seems to both get around the target file -vs- dir issue, and still gives the user choice for where to stage the transfer.

(Originally by Ashton Davis)

Comment 11 Aleksei Slaikovskii 2017-06-12 09:31:11 UTC
Verified on ovirt-engine-4.1.3.2-0.1.el7.noarch

# engine-backup --help | grep tmpdir                                                                                                         
--tmpdir=DIR   Set temporary directory parent. See also mktemp(1).

# mkdir /var/backup_tmpdir

# engine-backup --tmpdir=/var/backup_tmpdir --mode=backup --file=backup --log=backup.log
Backing up:
Notifying engine
- Files
- Engine database 'engine'
- DWH database 'ovirt_engine_history'
Packing into file 'backig'
Notifying engine
Done.

Comment 15 errata-xmlrpc 2017-07-06 07:30:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1692

Comment 16 Daniel Gur 2019-08-28 12:57:46 UTC
sync2jira

Comment 17 Daniel Gur 2019-08-28 13:02:56 UTC
sync2jira

Comment 18 Daniel Gur 2019-08-28 13:12:19 UTC
sync2jira

Comment 19 Daniel Gur 2019-08-28 13:16:31 UTC
sync2jira