Bug 1962905 - Ramdisk ISO source defaulting to "http" breaks deployment on a good amount of BMCs [NEEDINFO]
Summary: Ramdisk ISO source defaulting to "http" breaks deployment on a good amount of...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Bare Metal Hardware Provisioning
Version: 4.8
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.8.0
Assignee: Riccardo Pittau
QA Contact: Chad Crum
jfrye
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-05-20 18:33 UTC by Antoni Segura Puimedon
Modified: 2021-07-27 23:09 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Release Note text: Previously, the image caching mechanism in Ironic was disabled to enable a direct connection to the HTTP server that hosts the virtualmedial iso to prevent local storage issues. Non-standard compliant HTTP clients and redfish implementations caused failures on BMC connections. This has been fixed by reverting to the default Ironic behavior where the virtualmedia iso is cached and served from the Ironic conductor node. Issues caused by non-standard compliant HTTP clients and redfish implementations have been fixed. ------- Cause: The image caching mechanism in ironic was disabled in favor of direct connection to the http server that hosts the virtualmedia iso to prevent issues with local storage. Consequence: Failure on BMC connections due to non-standard compliant HTTP client and redfish implementations. Fix: Revert default ironic behavior to cache and serve the virtualmedia iso. Result: The default ironic behavior has been reinstated, the virtualmedia iso is cached and served from the ironic conductor node, issues with non-standard compliant HTTP client and redfish implementations has been fixed.
Clone Of:
Environment:
Last Closed: 2021-07-27 23:09:41 UTC
Target Upstream Version:
ccrum: needinfo? (asegurap)


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift ironic-image pull 172 0 None open Bug 1962905: Revert "Disable caching live boot iso by default" 2021-05-20 18:48:36 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 23:09:56 UTC

Description Antoni Segura Puimedon 2021-05-20 18:33:04 UTC
Description of problem:
In https://bugzilla.redhat.com/show_bug.cgi?id=1953979 "Ironic caching virtualmedia images results in disk space limitations" a new configuration setting was added to Ironic that allows it to set whether it should cache the images locally (while attached) or let the source URL be the one that the BMC reaches out to attach the virtualmedia ISO.

In the ticket discussion there was some debate as to whether we should expose that configurability to the Cluster Baremetal Operator or we should just make the default be the new "http" source. Finally the latter approach was taken, while the configurability might come at a later release.

While the new default does address the storage concerns, it reminded us that a significant amount of the BMC implementations in the field have non-standard compliant HTTP client and redfish implementations that call for revisiting the ramdisk ISO source default.

Ironic's approach to caching ISOs and serving them to Apache is well supported by most BMCs and defaulting to it would increase the compatibility of the solution. Switching the default to "local" should come with documentation of the storage requirements, specially in the usage pattern introduced by the Agent based fully automated provisioning.

Version-Release number of selected component (if applicable): 4.8

Comment 3 nshidlin 2021-06-29 17:35:21 UTC
this bug doesn't block OCP4.8.0 -release -  the fix will be delivered the the ACM channel on the AI-operator add verified then

Comment 7 Chad Crum 2021-07-13 13:37:23 UTC
Right - BMH is correct.

I double checked the ironic logs and I do see a cached version of the image getting posted to sushy tools api (http://192.168.123.86:6180/redfish/boot-40e51898-0bf2-4722-8400-cf5bec46d317.iso?filename=tmpu5bot05v.iso) - so this is validated (ACM DS 2.3.0-DOWNSTREAM-2021-07-12-15-34-40)


2021-07-12 20:24:53.539 1 DEBUG sushy.connector [req-826325b1-7ee6-47db-9437-0fb34aadbcfb ironic-user - - - -] HTTP request: POST https://192.168.123.1:8000/redfish/v1/Managers/8699f597-606b-4f18-afc3-039440271a4e/VirtualMedia/Cd/Actions/VirtualMedia.InsertMedia; headers: {'Content-Type': 'application/json', 'OData-Version': '4.0'}; body: {'Image': 'http://192.168.123.86:6180/redfish/boot-40e51898-0bf2-4722-8400-cf5bec46d317.iso?filename=tmpu5bot05v.iso', 'Inserted': True, 'WriteProtected': True}; blocking: False; timeout: 60; session arguments: {}; _op /usr/lib/python3.6/site-packages/sushy/connector.py:110ESC[00m
/usr/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
2021-07-12 20:24:54.261 1 DEBUG sushy.connector [req-826325b1-7ee6-47db-9437-0fb34aadbcfb ironic-user - - - -] HTTP response for POST https://192.168.123.1:8000/redfish/v1/Managers/8699f597-606b-4f18-afc3-039440271a4e/VirtualMedia/Cd/Actions/VirtualMedia.InsertMedia: status code: 204 _op /usr/lib/python3.6/site-packages/sushy/connector.py:184ESC[00m
2021-07-12 20:24:54.262 1 INFO ironic.drivers.modules.redfish.boot [req-826325b1-7ee6-47db-9437-0fb34aadbcfb ironic-user - - - -] Inserted boot media http://192.168.123.86:6180/redfish/boot-40e51898-0bf2-4722-8400-cf5bec46d317.iso?filename=tmpu5bot05v.iso into cd for node 40e51898-0bf2-4722-8400-cf5bec46d317ESC[00m

Comment 10 errata-xmlrpc 2021-07-27 23:09:41 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438


Note You need to log in before you can comment on or make changes to this bug.