(Filing on behalf of customer)

Summary: Engine backup currently stages data to /tmp, which in an emergency backup situation (such as a filling disk to due DWH overflow) is a problem, because it can fill the disk up and crash the manager.

Description:

$customer had a situation where the filesystem on HostedEngine was near 99% full with only 1.2GB left of space. They wanted to create an engine backup with scope=all and make the destination of the backup to an NFS share (since they didn't have space on my engine). The command they used for engine-backup is below (note that /engine-backup is an nfs mounted directory)

/usr/bin/engine-backup --mode=backup --scope=all --archive-compressor=gzip --file=/engine-backups/dbdump/EngineDBbkp-$(hostname)-$(date '+%Y%m%d.%H%M').gzip --log=/var/tmp/EngineDBbkp-$(date '+%Y%m%d.%H%M').log


Customer noticed it was writing to /tmp filesystem (through the backup log), which led it to filling up HostedEngine's filesystem to 100%.

The engine-backup should not stage files within /tmp, it should instead only utilize the location specified in the '--file=' parameter passed to engine-backup. If a staging directory is needed, perhaps add an option to specify the staging directory as well? 

You can see in the log below from the engine-backup that the target was /engine-backup directory but later in the log you see it staging all the files in a temporary directory in  /tmp/engine-backup.vhDTVpcYlG/



2017-04-04 08:31:59 1072: Start of engine-backup mode backup scope all file /engine-backups/dbdump/EngineDBbkp-rhvm-engine01.[redacted]-20170404.0831.gzip
2017-04-04 08:31:59 1072: OUTPUT: Backing up:
2017-04-04 08:31:59 1072: Generating pgpass
2017-04-04 08:31:59 1072: OUTPUT: Notifying engine
2017-04-04 08:31:59 1072: pg_cmd running: psql -w -U engine -h localhost -p 5432  engine -t -c SELECT LogEngineBackupEvent('files', now(), 0, 'Started', 'rhvm-engine01.[redacted]', '/var/tmp/EngineDBbkp-20170404.0831.log');


2017-04-04 08:31:59 1072: pg_cmd running: psql -w -U engine -h localhost -p 5432  engine -t -c SELECT LogEngineBackupEvent('db', now(), 0, 'Started', 'rhvm-engine01.[redacted]', '/var/tmp/EngineDBbkp-20170404.0831.log');


2017-04-04 08:31:59 1072: pg_cmd running: psql -w -U engine -h localhost -p 5432  engine -t -c SELECT LogEngineBackupEvent('dwhdb', now(), 0, 'Started', 'rhvm-engine01.[redacted]', '/var/tmp/EngineDBbkp-20170404.0831.log');


2017-04-04 08:31:59 1072: Creating temp folder /tmp/engine-backup.vhDTVpcYlG/tar
2017-04-04 08:31:59 1072: OUTPUT: - Files
2017-04-04 08:31:59 1072: Backing up files to /tmp/engine-backup.vhDTVpcYlG/tar/files
2017-04-04 08:32:01 1072: OUTPUT: - Engine database 'engine'
2017-04-04 08:32:01 1072: Backing up database to /tmp/engine-backup.vhDTVpcYlG/tar/db/engine_backup.db
2017-04-04 08:32:01 1072: pg_cmd running: pg_dump -w -U engine -h localhost -p 5432  engine -E UTF8 --disable-dollar-quoting --disable-triggers --format=custom
2017-04-04 08:32:05 1072: OUTPUT: - DWH database 'ovirt_engine_history'
2017-04-04 08:32:05 1072: Backing up dwh database to /tmp/engine-backup.vhDTVpcYlG/tar/db/dwh_backup.db
2017-04-04 08:32:05 1072: pg_cmd running: pg_dump -w -U ovirt_engine_history -h localhost -p 5432  ovirt_engine_history -E UTF8 --disable-dollar-quoting --disable-triggers --format=custom

Where are you experiencing the behavior?  What environment?

RHVM 4.0.6
Hypervisors: RHEL 7.3 (RHEV 4.0)

Reprodicibility: 
every time