Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1953979

Summary:	Ironic caching virtualmedia images results in disk space limitations
Product:	OpenShift Container Platform	Reporter:	Antoni Segura Puimedon <asegurap>
Component:	Bare Metal Hardware Provisioning	Assignee:	Riccardo Pittau <rpittau>
Bare Metal Hardware Provisioning sub component:	ironic	QA Contact:	Lubov <lshilin>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	high	CC:	ccrum, janders, keyoung, lshilin, rbartal, rfreiman
Version:	4.8	Keywords:	Triaged
Target Milestone:	---	Flags:	asegurap: needinfo-
Target Release:	4.8.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-07-27 23:04:05 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Antoni Segura Puimedon 2021-04-27 10:53:27 UTC

Description of problem:
In order to ensure ISO availability for BMC (both from the perspective of it being reachable and to make sure that it stays reachable throughout the provisioning process) Ironic downloads the images that it is to virtual-media attach and keeps them until the machine is un-deployed.

With usage patterns that needs to start many virtual ram-disks, ironic can fill the ephemeral storage it uses and cause the ironic pod to be evicted due to OCP resource usage handling

Version-Release number of selected component (if applicable):
4.8

How reproducible:


Steps to Reproduce:
1. Prepare an cdrom sized ISO and make it available via HTTP
2. Create a large pool of bare metal systems and the BMH for them
3. Set the ISO from step (1) to the image section of all the BMH image spec section for live-iso booting

Actual results:
Ironic batches the attachment of the ISOs to the nodes without any de-deduplication but eventually runs out of ephemeral storage.


Expected results:
All the systems boot the ISO from step (1)

Additional info:
If there was a way to disable caching, then the ISO URL provider would be responsible for the ISO being available and reachable during the lifetime of the virtual ram-disk OS.

Comment 1 Rom Freiman 2021-04-27 10:56:38 UTC

I would say it affects also:
1. consistency?
2. Doubles the load on the cluster because ironic has to fetch it from the AI operator, and then serve it.

Comment 2 Dmitry Tantsur 2021-04-27 16:39:56 UTC

Dev notes: we have image_download_source for the direct deploy already, but it's not respected in the ramdisk deploy (and the default is different from what we want). The current behavior corresponds to image_download_source=local, we need an option image_download_source=http (with the similar semantic to the direct deploy).

Then we need to update BMO to use the new option when the live ISO workflow is requested.

Comment 10 Chad Crum 2021-05-13 12:21:38 UTC

I can run through testing today...

Comment 14 errata-xmlrpc 2021-07-27 23:04:05 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438