Bug 1865944

Summary: OpenShift master nodes disk pressure
Product: OpenShift Container Platform Reporter: Jan Šafařík <jsafarik>
Component: InstallerAssignee: Mike Fedosin <mfedosin>
Installer sub component: OpenShift on OpenStack QA Contact: David Sanz <dsanzmor>
Status: CLOSED DUPLICATE Docs Contact:
Severity: unspecified    
Priority: unspecified CC: adahiya, aos-bugs, eparis, jokerman, pprinett
Version: 4.5Keywords: UpcomingSprint
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-08-11 17:56:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jan Šafařík 2020-08-04 15:12:36 UTC 
Description of problem:

We at Fuse Online have installed new OCP 4.5 (4.5.3) on OpenStack. We are using OSIA tool which does most of the installation work for us. After ~2 days of use, the clusters started to have problems to deploy pods due to disk pressure. When I tried to examine the nodes, I found out that there was an error stating: "wanted to free 1761579827 bytes, but freed 0 bytes space with errors in image deletion". I have tried to manually prune the images, some were deleted, but it didn't make much a difference and the disk pressure was still present. After another ~2 days (weekend), the nodes had the disk spaces 100% used, which resulted in not being able to even log into those OCP. According to the docs (https://docs.openshift.com/container-platform/4.5/installing/installing_openstack/installing-openstack-installer-custom.html#installation-osp-control-compute-machines_installing-openstack-installer-custom) we have met the minimum required storage space.

Is this a bug in the documents and should the disks be bigger than that (we have each master node with 40G of space)? Or could this be caused by something else I am missing?


Version-Release number of selected component (if applicable): 4.5.3
Comment 1 Abhinav Dahiya 2020-08-04 23:17:11 UTC 
> After another ~2 days (weekend), the nodes had the disk spaces 100% used, which resulted in not being able to even log into those OCP. According to the docs (https://docs.openshift.com/container-platform/4.5/installing/installing_openstack/installing-openstack-installer-custom.html#installation-osp-control-compute-machines_installing-openstack-installer-custom) we have met the minimum required storage space.


i think the docs mention minimum requirements, and how much space you need depends on users' setup. So i think planning is important and we shouldn't expect the minimum to take that into account.
Comment 2 Jan Šafařík 2020-08-05 07:36:55 UTC 
I understand that, the thing is that we have been using ci.m1.xlarge (which has 40G of disk space) for a long time, and it always has been enough. What more, manual image pruning did not free almost any space, so we weren't able to recover from the disk pressure even when the clusters were still accessible (and from that time, the OCPs have not been used).
Comment 7 Mike Fedosin 2020-08-11 17:56:14 UTC 

*** This bug has been marked as a duplicate of bug 1858498 ***