Bug 996491

Summary: Do not check quota explicitly when "snapshot" an app with large database which consumes 90% quota
Product: OpenShift Online Reporter: Qiushui Zhang <qiuzhang>
Component: ContainersAssignee: Dan Mace <dmace>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.xCC: dmcphers, hasari, xtian
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-08-29 12:50:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
an example to create large data file none

Description Qiushui Zhang 2013-08-13 09:41:10 UTC 
Created attachment 786095 [details]
an example to create large data file

Description of problem:
When trying to snapshot save an application with a big database files ( see it has consumed more than 900M) , it reports success  even if met disk quota  exceed error. but actually it did not save the whole database dump successfully after confirming it by restore the saved snapshot  back to another app.
Did not find any explict errors or warnings in  in /var/log/mcollective.log  either even if  the client message is blocked by bug https://bugzilla.redhat.com/show_bug.cgi?id=994174
Version-Release number of selected component (if applicable):
devenv_3640

How reproducible:
always

Steps to Reproduce:
1. Create an app with database added, e.g. rhc app create php1 php-5.3 mysql-5.1
2. SSH into the app. Fill the app with database data. An example could be sql script attached to the bug. Make sure that after this step, more than 90% space is used.
3. On instance, tailf ./var/log/mcollective.log
4. rhc snapshot save php1
5. Create another app. 
   rhc app create php2 php-5.3 mysql-5.1
6. Restore the file generated from the first app php1 to php2. 
   rhc snapshot restore -a php2 -f php1.tag.gz

Actual results:
[root@ip-10-164-47-156 ~]# rhc snapshot save php1
Pulling down a snapshot to php1.tar.gz...
gzip: stdout: Disk quota exceeded
mysqldump: Got errno 32 on write
Couldn't not dump mysql! Continuing anyway
Creating and sending tar.gz
RESULT:
Success
1). It shows "Disk quota exceeded" although it ends up "success".
2). If restoring this archive to other app (e.g php2 with mysql), the database will not be restored.
3). No logs is added to /var/log/mcollective.log.


Expected results:
Quota check should be done when snapshot an app. If the quota is used more than 90%(or other percentage we want to inform the user), a warning message should be given.


Additional info:
Comment 1 Dan Mace 2013-08-20 13:35:01 UTC 
https://github.com/openshift/origin-server/pull/3427
Comment 2 openshift-github-bot 2013-08-21 18:01:34 UTC 
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/a55be7528dd3063aad988cb7c864a56a9164b34e
Bug 996491: Show warning when using snapshot and near disk quota

When the gear user's disk usage is >= 90% of quota, display a warning when
snapshotting or restoring (as these operations can fail in unexpected ways
when disk writes fail).
Comment 3 Qiushui Zhang 2013-08-22 02:34:39 UTC 
Tested on devenv_3683.
When "snapshot save" an app with more than 90% storage used, I got the following warning message:
"WARNING: The application's disk usage is very close to the quota limit. The snapshot may fail unexpectedly"


When "restore", I got a warning like:
"WARNING: The application's disk usage is very close to the quota limit. The restore may fail unexpectedly"

Mark the bug as verified.