Description of problem: The IPA doesn't successfully finish the introspection of bare metal servers that dont have local disks attached. The root disks for these servers is provided through the iSCSI LUN(s) hosted on a storage array. Version-Release number of selected component (if applicable): OSP 10 How reproducible: Invoke introspection of the baremetal servers while passing the "rd.iscsi.ibft=1 and rd.iscsi.firmware=1" parameters for kernel command line in /httpboot/inspector.ipxe Steps to Reproduce: 1.Start the introspection with openstack baremetal introspection start <node-id> Additional info: Information from ramdisk logs Aug 14 14:59:44 localhost.localdomain ironic-python-agent[1275]: Command: iscsistart -f Aug 14 14:59:44 localhost.localdomain ironic-python-agent[1275]: Exit code: 21 Aug 14 14:59:44 localhost.localdomain ironic-python-agent[1275]: Stdout: u'Could not get list of targets from firmware.\n' Kernel command line parmeters for IPA ramdisk: Aug 14 14:58:38 localhost.localdomain kernel: Command line: ipa-inspection-callback-url=http://172.21.82.221:5050/v1/continue ipa-inspection-collectors=default,extra-hardware,logs systemd.journald.forward_to_console=yes BOOTIF=00:25:b5:00:0a:11 ipa-debug=1 ipa-inspection-dhcp-all-interfaces=1 ipa-collect-lldp=1 rd.iscsi.ibft=1 rd.iscsi.firmware=1 initrd=agent.ramdisk Loads the iBFT plugin Aug 14 14:59:12 localhost.localdomain NetworkManager[875]: <info> [1502737152.4168] settings: loaded plugin iBFT: (c) 2014 Red Hat, Inc. But I dont see the iBFT interfaces configured from ip_addr file ibft0 and ibft1 are missing. Typically it should read the iBFT table, pull information and configure the iBFT interfaces.
Related Ironic versions: openstack-ironic-api.noarch 1:6.2.3-1.el7ost openstack-ironic-common.noarch 1:6.2.3-1.el7ost openstack-ironic-conductor.noarch 1:6.2.3-1.el7ost openstack-ironic-inspector.noarch 4.2.1-3.el7ost openstack-ironic-python-agent.noarch 1.5.1-3.el7ost
Hi! Redirecting this to diskimage-builder and to the appropriate team. We cannot patch ironic-python-agent, because when it starts it's already too late to configure networking. We do include the 'iscsistart' hack to make sure the disks are connected, but this assumes working networking anyway. Maybe this problem should be fixed in dracut instead. I'll leave it up for DIB folks to decide.
Do you have the IPA boot logs handy, so we can check if the interfaces are being configured properly?
Created attachment 1314174 [details] IPA boot log
I believe the iBFT interfaces are configured from the BIOS information. (Not from DHCP). I am suspecting a kernel module is missing in the image which might configure these.
Do you know which is the name of the iBFT interfaces? Can you confirm that you have link to it?
From the ip_addr file from the ramdisk logs, I see that enp7s0 and enp8s0 are the iBFT interfaces. Typically these should be configured from BIOS and should be renamed to ibft0 and ibft1 respectively. (The links are Up).
Can you patch manually the following code? http://git.openstack.org/cgit/openstack/ironic-python-agent/tree/ironic_python_agent/hardware.py?h=stable/newton#n88 And replace 'iscsistart -f' with 'iscsistart --debug -f' ? So we can get more useful output from the iscsistart command.
Actually iscsistart --debug=1 -f
Let me try putting in an debug option there. But what is making me nervous is, the iBFT interfaces arent getting configured or in other words the interfaces arent getting source IP etc from initiator Boot Firmware table.
I tried adding the debug=1 patch, didnt yield any substantial info though Aug 18 12:55:53 localhost.localdomain ironic-python-agent[1125]: Command: iscsistart --debug=1 -f Aug 18 12:55:53 localhost.localdomain ironic-python-agent[1125]: Exit code: 21 Aug 18 12:55:53 localhost.localdomain ironic-python-agent[1125]: Stdout: u'Could not get list of targets from firmware.\n' Aug 18 12:55:53 localhost.localdomain ironic-python-agent[1125]: Stderr: u'' _check_for_iscsi /usr/share/ironic-python-agent/venv/lib/python2.7/site-packages/ironic_python_agent/hardware.py:91
Added a fix in master: https://review.openstack.org/499082
The code merged for Queens. I suggest cloning this bug, if the fix is needed in earlier versions (it will have to happen downstream, as DIB is branchless).
Installed lastest osp 13 and verify code in present. we don't have specific hardware to test but our development reports the originator of bug has verified with his NetApp hardware. Env: openstack-tripleo-image-elements-8.0.1-1.el7ost.noarch Verified code landed and files which were removed /usr/share/tripleo-image-elements/ironic-agent-multipath/element-deps install-static /usr/share/tripleo-image-elements/ironic-agent-multipath/static/etc/modules-load.d/iscsi.conf qla4xxx cxgb3i cxgb4i bnx2i be2iscsi iscsi_boot_sysfs iscsi_ibft iscsi_tcp target_core_mod /usr/share/tripleo-image-elements/ironic-agent-multipath/README.rst ====================== ironic-agent-multipath ====================== Updates the ironic agent, installing multipath and iscsi package, and enabling needed modules by default, to execute a modprobe for the needed drivers before it is started.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:2086