Description of problem:

We are observing intermittent deploy failures on Dell machines caused by virtual media failing to attach. An example of this (at the inspection stage):

Failed to inspect hardware. Reason: unable to start inspection: HTTP POST https://10.19.0.84/redfish/v1/Managers/iDRAC.Embedded.1/VirtualMedia/CD/Actions/VirtualMedia.InsertMedia returned code 500. Base.1.2.GeneralError: The request failed due to an internal service error.  The service is still operational. Extended information: [{'Message': 'The request failed due to an internal service error.  The service is still operational.', 'MessageArgs': [], 'MessageArgs': 0, 'MessageId': 'Base.1.2.InternalError', 'RelatedProperties': [], 'RelatedProperties': 0, 'Resolution': 'Resubmit the request.  If the problem persists, consider resetting the service.', 'Severity': 'Critical'}]

Version-Release number of selected component (if applicable):

OpenShift 4.8

How reproducible:

The issue isn't easily reproducible, but it happens regularly. It seems correlated with pending jobs stuck in Lifecycle controller. On a high level, it could be described as iDRAC ending up in corrupt state.

Clearing Lifecycle Controller jobs and resetting iDRAC seems to reliably resolve this issue.


Steps to Reproduce:
1.
2.
3.

Actual results:

Virtual media based deployment fails.


Expected results:


Virtual media based deployment succeeds.

Additional info:

Work to address this in Ironic has been undertaken upstream by both Dell and Red Hat. There is existing Ironic code contributed by Dell which can be used to automatically clear Lifecycle Controller jobs and reset iDRAC prior to deployment to ensure a known good state. This is essentially a programmatic way of applying the exact same fix we have been applying manually so far to help resolve this issue when it's observed.  Reference upstream stories are linked to the BZ.