Bug 1865944 - OpenShift master nodes disk pressure
Summary: OpenShift master nodes disk pressure
Keywords:
Status: CLOSED DUPLICATE of bug 1858498
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.5
Hardware: Unspecified
OS: Linux
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Mike Fedosin
QA Contact: David Sanz
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-08-04 15:12 UTC by Jan Šafařík
Modified: 2020-08-11 17:56 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-08-11 17:56:14 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Jan Šafařík 2020-08-04 15:12:36 UTC
Description of problem:

We at Fuse Online have installed new OCP 4.5 (4.5.3) on OpenStack. We are using OSIA tool which does most of the installation work for us. After ~2 days of use, the clusters started to have problems to deploy pods due to disk pressure. When I tried to examine the nodes, I found out that there was an error stating: "wanted to free 1761579827 bytes, but freed 0 bytes space with errors in image deletion". I have tried to manually prune the images, some were deleted, but it didn't make much a difference and the disk pressure was still present. After another ~2 days (weekend), the nodes had the disk spaces 100% used, which resulted in not being able to even log into those OCP. According to the docs (https://docs.openshift.com/container-platform/4.5/installing/installing_openstack/installing-openstack-installer-custom.html#installation-osp-control-compute-machines_installing-openstack-installer-custom) we have met the minimum required storage space.

Is this a bug in the documents and should the disks be bigger than that (we have each master node with 40G of space)? Or could this be caused by something else I am missing?


Version-Release number of selected component (if applicable): 4.5.3

Comment 1 Abhinav Dahiya 2020-08-04 23:17:11 UTC
> After another ~2 days (weekend), the nodes had the disk spaces 100% used, which resulted in not being able to even log into those OCP. According to the docs (https://docs.openshift.com/container-platform/4.5/installing/installing_openstack/installing-openstack-installer-custom.html#installation-osp-control-compute-machines_installing-openstack-installer-custom) we have met the minimum required storage space.


i think the docs mention minimum requirements, and how much space you need depends on users' setup. So i think planning is important and we shouldn't expect the minimum to take that into account.

Comment 2 Jan Šafařík 2020-08-05 07:36:55 UTC
I understand that, the thing is that we have been using ci.m1.xlarge (which has 40G of disk space) for a long time, and it always has been enough. What more, manual image pruning did not free almost any space, so we weren't able to recover from the disk pressure even when the clusters were still accessible (and from that time, the OCPs have not been used).

Comment 7 Mike Fedosin 2020-08-11 17:56:14 UTC

*** This bug has been marked as a duplicate of bug 1858498 ***


Note You need to log in before you can comment on or make changes to this bug.