Red Hat Bugzilla – Bug 849217
Disk overcommit maybe happended when allow user to set quota
Last modified: 2015-05-14 19:16:51 EDT
Description of problem: Now REST API for seting qutoa for gear group is completed, but it maybe cause disk overcomit. Currently the total availabe free disk space for gear is about 7G in instance # df -h Filesystem Size Used Avail Use% Mounted on /dev/xvde1 7.6G 4.4G 3.1G 59% / none 1.8G 0 1.8G 0% /dev/shm /dev/xvde2 7.6G 1.4G 6.1G 19% /var User create an app, and set quota to 30G. $ curl -k -X PUT -H 'Accept: application/xml' --user jialiu@redhat.com:214214 -d storage=30 https://ec2-50-16-130-0.compute-1.amazonaws.com/broker/rest/domains/jialiu/applications/phptest/gear_groups/49523e0fdb9944968fbd0bde4af77ed6/resources <?xml version="1.0" encoding="UTF-8"?> <response> <status>ok</status> <messages/> <version>1.1</version> <type>resources</type> <supported-api-versions> <supported-api-version>1.0</supported-api-version> <supported-api-version>1.1</supported-api-version> </supported-api-versions> <data> <resources> <uuid>49523e0fdb9944968fbd0bde4af77ed6</uuid> <storage>30</storage> </resources> </data> </response> But actually this user never can write 30G data in his app. There should be some solution to avoid such problem. Version-Release number of selected component (if applicable): devenv_2015 How reproducible: Always Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
This bug is tricky. My gut reaction is this isn't a bug because operations wants this setup so they can overcommit. However, we might need to take more seriously when we are overcommiting storage users are paying for. So I think the rule we have to follow is no single request won't have enough space when the additional gear is added. So, for example, 20 gears can all be created that have a quota of 30 gigs as long as when each gear is added there is at least 30 gigs left. Transferring to the runtime team to get agreement and raise an error on the case mentioned.
I don't believe temporary overcommit is a problem, but it does open up a deeper issue with paid applications. We should consider balancing storage usage along with gear count when placing gears. That's an easy decision and it doesn't disrupt anything that's active. Once a gear has been placed, if the owner is paying us for more storage on it we can't really say no within the limits of what was advertised. If there isn't enough space then something has to move to accommodate it. Having to migrate someone who is busy sucks, especially if they are paying us for high uptime in addition to more storage. Idle gears, especially on free apps are the least risk to move. We may want to periodically (daily? every 4 hours?) re-decide placement for idle gears and move them to balance out storage allocation.
Online monitors and corrections before this issue can arise.