Bug 691557

Summary: need check for fs space before building a image
Product: [Retired] CloudForms Cloud Engine Reporter: Dave Johnson <dajohnso>
Component: imagefactoryAssignee: Steve Loranz <sloranz>
Status: CLOSED CURRENTRELEASE QA Contact: Dave Johnson <dajohnso>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 0.3.1CC: clalance, cpelland, deltacloud-maint, nobody, whayutin
Target Milestone: beta   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-06-14 16:11:24 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 684278    

Description Dave Johnson 2011-03-28 20:05:39 UTC
Ran into a issue where the web ui stopped working after I built three images at once.  I checked the file system space and / was at 100% usage.  Specifically the /var/lib and /var/tmp were consuming all the available space.  

We need some logic that determines if the system can allocate another image safely and still have room to breathe.

Comment 1 Chris Lalancette 2011-03-28 20:19:05 UTC
Interesting observation.  There are actually some checks in Oz that see if there is enough disk space to perform the prep work, but there aren't further checks to make sure that the installation won't fill up / (mostly because we don't know).  Do you have any idea where most of the space was used up?

Chris Lalancette

Comment 2 Steve Loranz 2011-03-28 21:03:07 UTC
In order to really do this correctly, the builders would have to reserve space with a disk space manager object.  The only way to do that correctly is to prefetch the sizes of all of the packages that are listed in the template.  My gut reaction to this is that it's doable but not trivial.

As an interim solution, to avoid developer system falling over from filled up root partitions, how about we add options to not cache the iso files and images that are produced once they are stored in warehouse?  This helps but will not eliminate the problem if one has iwhd running locally and storing data in /.

-steve

Comment 3 Dave Johnson 2011-03-29 17:38:38 UTC
Chris, /var/lib was the bigger offender of the two directories mention comment 0


[root@ibm-ls42-01 var]# df -H
Filesystem             Size   Used  Avail Use% Mounted on
/dev/mapper/vg_ibmls4201-lv_root
                        35G    31G   2.5G  93% /
tmpfs                  2.1G      0   2.1G   0% /dev/shm
/dev/sda1              508M    30M   452M   7% /boot
/dev/mapper/vg_ibmls4201-lv_home
                        31G   181M    30G   1% /home
[root@ibm-ls42-01 var]# du -h /var --max-depth=1
12K	./yp
256K	./run
8.0K	./empty
5.5G	./tmp
444K	./beah
4.0K	./opt
4.0K	./cvs
4.0K	./account
21G	./lib
4.0K	./preserve
279M	./cache
4.0K	./games
16K	./db
2.4M	./log
4.0K	./local
28K	./lock
1.2M	./www
60K	./spool
4.0K	./nis
26G	.

Comment 4 Steve Loranz 2011-03-30 16:54:52 UTC
I disagree that this is a blocker for you guys.

Comment 5 wes hayutin 2011-03-30 17:15:52 UTC
agreed this is not a blocker for testing... but QE would like it fixed for the release before summit.

Comment 6 wes hayutin 2011-12-08 14:14:58 UTC
perm close