Bug 1286703

Summary: Introspection fails while installing Liberty on certain hardware (DELL PowerEdge R710)
Product: Red Hat OpenStack Reporter: Raoul Scarazzini <rscarazz>
Component: rhosp-directorAssignee: chris alfonso <calfonso>
Status: CLOSED NOTABUG QA Contact: yeylon <yeylon>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 8.0 (Liberty)CC: dtantsur, hbrock, liron, mburns, michele, rhel-osp-director-maint, srevivo
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-11-30 16:52:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
DHCP failure screenshot
none
IPA data screenshot none

Description Raoul Scarazzini 2015-11-30 14:31:18 UTC
Description of problem:

Installing Liberty via OSPd ends in error on DELL PowerEdge R710, introspection fails, it seems because DHCP fails on all interfaces. Maybe the problem is related to the InfiniBan adapters.

Version-Release number of selected component (if applicable):

8-director (rhos release)

How reproducible:

Always.

Steps to Reproduce:
1. Configure repos with rhos-release -P 8-director (using RHEL 7.2 as undercloud);
2. Generate images:

export USE_DELOREAN_TRUNK=0
export RHOS=1
export DIB_LOCAL_IMAGE=rhel-guest-image-7.2-20151102.0.x86_64.qcow2
export DIB_YUM_REPO_CONF="/etc/yum.repos.d/rhos-release-8.repo  /etc/yum.repos.d/rhos-release-rhel-7.2.repo /etc/yum.repos.d/rhos-release-8-director.repo"
openstack overcloud image build --all

3. Launch introspection:

openstack overcloud image upload --image-path /home/stack/images/
json_verify < instackenv.json
openstack baremetal import --json ~/instackenv.json
openstack baremetal configure boot
openstack baremetal introspection bulk start

Actual results:

Introspection didn't finish for nodes 1b652cd0-4738-45a3-9528-218203c1ffca,2f6d1b7d-50e5-4bf7-9a89-92c549729dbf,341dd84f-3016-4ade-a171-51b76dca335e,f488d010-9f53-494f-832e-d4288df0ffe3

Expected results:

Introspection completed without any error.

Additional info:

The same hardware gets installed with the release 7 of the director.

Comment 2 Dmitry Tantsur 2015-11-30 14:43:40 UTC
Created attachment 1100515 [details]
DHCP failure screenshot

The first thing that I've noticed on the target machine is failed dhcp-all-interfaces element, so I suspect it might be a DIB issue actually.

Comment 3 Dmitry Tantsur 2015-11-30 14:45:21 UTC
Created attachment 1100517 [details]
IPA data screenshot

The 2nd screenshot seems to confirm my guess. Don't pay attention to ERROR-level log messages, see the red arrow instead.

I suspect that no interfaces got any IP address, hence IPA could not possibly reach out to ironic-inspector.

Comment 4 Raoul Scarazzini 2015-11-30 16:52:05 UTC
We finally find out what was causing the problem. Even if the error messages were related to infiniband device, there were others related to the firmware of the bnx2 ethernet adapter, which were missing.

So it was sufficient to upload these files into the image:

modinfo -F firmware bnx2
bnx2/bnx2-rv2p-09ax-6.0.17.fw
bnx2/bnx2-rv2p-09-6.0.17.fw
bnx2/bnx2-mips-09-6.2.1b.fw
bnx2/bnx2-rv2p-06-6.0.15.fw
bnx2/bnx2-mips-06-6.2.3.fw

and, after regenerating the images, the introspection ended up successfully.

Closing this beacuse, in my opinion, this is not a bug.

Comment 5 Liron Cohen 2018-11-22 12:18:13 UTC
Please note that running the CentOS TripleO on R710 here is how we solved the problem:

After installing the undercloud and before the creation of the images
We edited the following file:
/usr/share/openstack-tripleo-common/image-yaml/overcloud-images.yaml

And added on both images the package:
linux-firmware

Then we run the build and upload commands:
openstack overcloud image build
--and--
openstack overcloud image upload


This solved the issue

Note:
Remember to run:
sudo -E tripleo-repos -b <VER> current
export DIB_YUM_REPO_CONF="/etc/yum.repos.d/delorean*"

And if you already created the images please remember to delete:
/httpboot/agent.kernel
/httpboot/agent.ramdisk
And the image upload will recreate them.