Bug 1312187 - Ironic fails to find the disk to write image to
Ironic fails to find the disk to write image to
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-ironic-python-agent (Show other bugs)
8.0 (Liberty)
x86_64 Unspecified
unspecified Severity unspecified
: ga
: 8.0 (Liberty)
Assigned To: Dmitry Tantsur
Raviv Bar-Tal
Depends On:
  Show dependency treegraph
Reported: 2016-02-25 22:07 EST by Sai Sindhur Malleni
Modified: 2016-05-02 10:24 EDT (History)
8 users (show)

See Also:
Fixed In Version: openstack-ironic-python-agent-1.1.0-6.el7ost
Doc Type: Bug Fix
Doc Text:
Sometimes, hard drives were not available in time for a deployment ramdisk run. Consequently, the deployment failed if the ramdisk was unable to find the required root device. With this update, the "udev settle" command is executed before enumerating disks in the ramdisk, and the deployment no longer fails due to the missing root device.
Story Points: ---
Clone Of:
Last Closed: 2016-04-07 17:31:26 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

External Trackers
Tracker ID Priority Status Summary Last Updated
OpenStack gerrit 284796 None None None 2016-02-29 11:21 EST
Red Hat Product Errata RHEA-2016:0603 normal SHIPPED_LIVE Red Hat OpenStack Platform 8 Enhancement Advisory 2016-04-07 20:53:53 EDT

  None (edit)
Description Sai Sindhur Malleni 2016-02-25 22:07:42 EST
Description of problem:
Overcloud deploy always fails after retries timing out. In a 3 controller+2 compute deployment at the most 4 of the nodes deployed before the stack create fails. Each time the nodes that fail to deploy are not necessarilythe same. Introspection completes without any complaints but in the ironic debug logs the following error is seen frequently. Some nodes which show the error also go through installation in subsequent attempts but some fail after retries. 

Failed to deploy instance: Failed to start the iSCSI target to deploy the node 79aa5db0-df3a-4f2e-9e6b-45820b18a97c. Error: {u'message': u'Error finding the disk or partition device to deploy the image onto: No suitable device was found for deployment - root device hints were not provided and all found block devices are smaller than 4294967296B [].', u'code': 404, u'type': u'DeviceNotFound', u'details': u'No suitable device was found for deployment - root device hints were not provided and all found block devices are smaller than 4294967296B [].'}

The systems  are PowerEdge R610 with 500gigs hard disk and as said introspection doesnt fail.


How reproducible: Always on the environment I tried

Steps to Reproduce:
1. Build undercloud
2.Finish introspection
3. Deploy overcloud

Actual results:Overcloud deploys

Expected results: Deploy fails

Additional info:
Using the latest OSP 8 bits. Local_boot option wasn't used
Comment 2 Lucas Alvares Gomes 2016-02-29 10:26:17 EST
So I've tested the ramdisk and apparently the problem is just the time that the IPA service starts.

I've first tried to modify the openstack-ironic-python-agent.service in the ramdisk to Require the systemd-udev-settle.service [0] service, but that didn't work.

Modifying the ramdisk manually to trigger udevadm and then settle in the code actually worked for me, e.g:

# udevadm trigger --verbose --dry-run --type=devices --subsystem-match=scsi_disk
# udevadm settle

I will work on a patch for IPA to do it.

[0] https://github.com/systemd/systemd/blob/master/units/systemd-udev-settle.service.in
Comment 3 Lucas Alvares Gomes 2016-03-02 12:24:40 EST
Fix was approved upstream and backported [0]

[0] https://code.engineering.redhat.com/gerrit/#/c/68938/
Comment 6 Raviv Bar-Tal 2016-04-07 04:05:01 EDT
Hi Sindhur,
I don't really know how to verify this bug,
There is not enough information and hw specification.

Can you tell if the fix did solve the problem for you?
If so I'll verify we have the current package in the release and close this bug.

Comment 7 Sai Sindhur Malleni 2016-04-07 08:54:25 EDT
Raviv Bar-Tal,

I tried the latest puddle/poodle and did not hit the issue, in fact in the performance team we have several working OSP 8 deployments, and haven't seen this issue after the fix.
Comment 8 errata-xmlrpc 2016-04-07 17:31:26 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.