Bug 1481806 - Introspection of Diskless servers (iSCSI Booted)
Summary: Introspection of Diskless servers (iSCSI Booted)
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-image-elements
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Linux
high
high
Target Milestone: Upstream M1
: 13.0 (Queens)
Assignee: Yolanda Robla
QA Contact: mlammon
URL:
Whiteboard:
Depends On:
Blocks: 1335596 1481831 1558679
TreeView+ depends on / blocked
 
Reported: 2017-08-15 18:54 UTC by Amit
Modified: 2018-09-13 17:09 UTC (History)
14 users (show)

Fixed In Version: openstack-tripleo-image-elements-8.0.0-0.20180117094122.02d0985.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1558679 (view as bug list)
Environment:
Last Closed: 2018-06-27 13:33:53 UTC
Target Upstream Version:


Attachments (Terms of Use)
IPA boot log (272.43 KB, text/x-vhdl)
2017-08-16 14:03 UTC, Amit
no flags Details


Links
System ID Priority Status Summary Last Updated
OpenStack gerrit 499082 'None' MERGED Update ironic-agent-multipath element 2020-08-28 06:58:21 UTC
Red Hat Product Errata RHEA-2018:2086 None None None 2018-06-27 13:34:45 UTC

Internal Links: 1496693 1496696 1496698 1496699

Description Amit 2017-08-15 18:54:29 UTC
Description of problem:
The IPA doesn't successfully finish the introspection of bare metal servers that dont have local disks attached. The root disks for these servers is provided through the iSCSI LUN(s) hosted on a storage array.

Version-Release number of selected component (if applicable):
OSP 10 

How reproducible:
Invoke introspection of the baremetal servers while passing the "rd.iscsi.ibft=1 and rd.iscsi.firmware=1" parameters for kernel command line in /httpboot/inspector.ipxe

Steps to Reproduce:
1.Start the introspection with openstack baremetal introspection start <node-id>

Additional info:
Information from ramdisk logs

Aug 14 14:59:44 localhost.localdomain ironic-python-agent[1275]: Command: iscsistart -f
Aug 14 14:59:44 localhost.localdomain ironic-python-agent[1275]: Exit code: 21
Aug 14 14:59:44 localhost.localdomain ironic-python-agent[1275]: Stdout: u'Could not get list of targets from firmware.\n'

Kernel command line parmeters for IPA ramdisk:
Aug 14 14:58:38 localhost.localdomain kernel: Command line: ipa-inspection-callback-url=http://172.21.82.221:5050/v1/continue ipa-inspection-collectors=default,extra-hardware,logs systemd.journald.forward_to_console=yes BOOTIF=00:25:b5:00:0a:11 ipa-debug=1 ipa-inspection-dhcp-all-interfaces=1 ipa-collect-lldp=1 rd.iscsi.ibft=1 rd.iscsi.firmware=1 initrd=agent.ramdisk

Loads the iBFT plugin
Aug 14 14:59:12 localhost.localdomain NetworkManager[875]: <info>  [1502737152.4168] settings: loaded plugin iBFT: (c) 2014 Red Hat, Inc.

But I dont see the iBFT interfaces configured from ip_addr file
ibft0 and ibft1 are missing.

Typically it should read the iBFT table, pull information and configure the iBFT interfaces.

Comment 1 Amit 2017-08-15 20:10:04 UTC
Related Ironic versions:

openstack-ironic-api.noarch          1:6.2.3-1.el7ost        
openstack-ironic-common.noarch       1:6.2.3-1.el7ost        
openstack-ironic-conductor.noarch    1:6.2.3-1.el7ost        
openstack-ironic-inspector.noarch    4.2.1-3.el7ost          
openstack-ironic-python-agent.noarch 1.5.1-3.el7ost

Comment 2 Dmitry Tantsur 2017-08-16 10:27:18 UTC
Hi!

Redirecting this to diskimage-builder and to the appropriate team. We cannot patch ironic-python-agent, because when it starts it's already too late to configure networking. We do include the 'iscsistart' hack to make sure the disks are connected, but this assumes working networking anyway.

Maybe this problem should be fixed in dracut instead. I'll leave it up for DIB folks to decide.

Comment 3 Yolanda Robla 2017-08-16 10:49:57 UTC
Do you have the IPA boot logs handy, so we can check if the interfaces are being configured properly?

Comment 5 Amit 2017-08-16 14:03:03 UTC
Created attachment 1314174 [details]
IPA boot log

Comment 6 Amit 2017-08-16 14:05:47 UTC
I believe the iBFT interfaces are configured from the BIOS information. (Not from DHCP). I am suspecting a kernel module is missing in the image which might configure these.

Comment 7 Yolanda Robla 2017-08-16 14:17:55 UTC
Do you know which is the name of the iBFT interfaces? Can you confirm that you have link to it?

Comment 8 Amit 2017-08-16 14:38:13 UTC
From the ip_addr file from the ramdisk logs, I see that enp7s0 and enp8s0 are the iBFT interfaces. Typically these should be configured from BIOS and should be renamed to ibft0 and ibft1 respectively. (The links are Up).

Comment 9 Yolanda Robla 2017-08-16 15:15:56 UTC
Can you patch manually the following code? 

http://git.openstack.org/cgit/openstack/ironic-python-agent/tree/ironic_python_agent/hardware.py?h=stable/newton#n88

And replace 'iscsistart -f' with 'iscsistart --debug -f' ? So we can get more useful output from the iscsistart command.

Comment 10 Yolanda Robla 2017-08-16 15:29:13 UTC
Actually iscsistart --debug=1 -f

Comment 12 Amit 2017-08-18 14:31:12 UTC
Let me try putting in an debug option there. But what is making me nervous is, the iBFT interfaces arent getting configured or in other words the interfaces arent getting source IP etc from initiator Boot Firmware table.

Comment 13 Amit 2017-08-18 17:07:30 UTC
I tried adding the debug=1 patch, didnt yield any substantial info though
Aug 18 12:55:53 localhost.localdomain ironic-python-agent[1125]: Command: iscsistart --debug=1 -f
Aug 18 12:55:53 localhost.localdomain ironic-python-agent[1125]: Exit code: 21
Aug 18 12:55:53 localhost.localdomain ironic-python-agent[1125]: Stdout: u'Could not get list of targets from firmware.\n'
Aug 18 12:55:53 localhost.localdomain ironic-python-agent[1125]: Stderr: u'' _check_for_iscsi /usr/share/ironic-python-agent/venv/lib/python2.7/site-packages/ironic_python_agent/hardware.py:91

Comment 14 Yolanda Robla 2017-09-02 07:00:23 UTC
Added a fix in master: https://review.openstack.org/499082

Comment 15 Dmitry Tantsur 2017-10-02 11:47:41 UTC
The code merged for Queens. I suggest cloning this bug, if the fix is needed in earlier versions (it will have to happen downstream, as DIB is branchless).

Comment 19 mlammon 2018-05-17 15:06:17 UTC
Installed lastest osp 13 and verify code in present.  we don't have specific hardware to test but our development reports the originator of bug has verified with his NetApp hardware.

Env:
openstack-tripleo-image-elements-8.0.1-1.el7ost.noarch

Verified code landed and files which were removed

/usr/share/tripleo-image-elements/ironic-agent-multipath/element-deps
install-static

/usr/share/tripleo-image-elements/ironic-agent-multipath/static/etc/modules-load.d/iscsi.conf
qla4xxx
cxgb3i
cxgb4i
bnx2i
be2iscsi
iscsi_boot_sysfs
iscsi_ibft
iscsi_tcp
target_core_mod

/usr/share/tripleo-image-elements/ironic-agent-multipath/README.rst
======================
ironic-agent-multipath
======================
Updates the ironic agent, installing multipath and
iscsi package, and enabling needed modules by default,
to execute a modprobe for the needed drivers before it is started.

Comment 27 errata-xmlrpc 2018-06-27 13:33:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086


Note You need to log in before you can comment on or make changes to this bug.