Bug 696267

Summary: [Broadcom 6.1 feat] Support bnx2i hba-mode and non-hba mode for boot in iscsi tools
Product: Red Hat Enterprise Linux 6 Reporter: Mike Christie <mchristi>
Component: iscsi-initiator-utilsAssignee: Andy Grover <agrover>
Status: CLOSED ERRATA QA Contact: Barry Donahue <bdonahue>
Severity: unspecified Docs Contact:
Priority: urgent    
Version: 6.1CC: akozumpl, bdonahue, coughlan, gideonn, jmir, mchan, mchristi, mzywusko, shiyer, syeghiay
Target Milestone: rcKeywords: FutureFeature, Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
: 696275 702211 (view as bug list) Environment:
Last Closed: 2011-05-19 14:14:59 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 696275, 702211    
Attachments:
Description Flags
output of iscsistart -b -d 8 command
none
non-HBA DVD installation steps none

Description Mike Christie 2011-04-13 17:59:11 UTC 
Description of problem:

From bz 442980:

We found out why it was failing on our side.  Please refer to my earlier
comment #72.  In order for the boot firmware to create the standard iBFT table,
we have to disable "HBA mode" during BIOS boot.  The side effect of this is
that the network MAC address will be used by the iSCSI boot firmware and this
same local MAC address will be put into the iBFT table.

I think anaconda uses this MAC address for the iface.hwaddress in the bnx2i
iface file.  This unfortunately does not match the iSCSI MAC address for the
bnx2i offload interface.

If we set it to HBA mode during boot, the iSCSI MAC address will be used, but
the iBFT will not have the standard signature and anaconda won't find it.

The iSCSI boot team made an experimental boot image to use standard iBFT
signature in HBA mode, and that made everything work.

So Ales, how did you get it to work on the 5708?  And how do we solve this
problem?

Thanks.
Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:
Comment 2 Mike Christie 2011-04-13 18:04:04 UTC 
Comment from Michael Chan on possible solution

> 1. Modify iscsi_ibft so it checks for BIFT. We can add some sort of callouts or
> driver override or something like we normally do, so drivers can hook in
> somehow.
> 

The search is done in userspace, right?  So send NETLINK to driver?

Alternatively, we already can get the ISCSI_HOST_PARAM_HWADDRESS from the
driver.  Can we just match that against the iBFT's MAC address?  If it matches,
we do offload.  If it doesn't match, we use iscsi_tcp.  We still need userspace
to add a search for the BIFT signature.
Comment 3 Mike Christie 2011-04-13 18:12:26 UTC 
(In reply to comment #2)
> Comment from Michael Chan on possible solution
> 
> > 1. Modify iscsi_ibft so it checks for BIFT. We can add some sort of callouts or
> > driver override or something like we normally do, so drivers can hook in
> > somehow.
> > 
> 
> The search is done in userspace, right?  So send NETLINK to driver?

I actually just meant that bnx2i would hook into iscsi_ibft/iscsi_ibft_find.c and tell it to look for your hba mode sig. Then if found iscsi_boot_sysfs would have some new file /sys/firmware/ibft/ethernet/offload that would print "yes" in this case.

iscsi tools would then check that and if set use bnx2i if not use iscsi_tcp.

> 
> Alternatively, we already can get the ISCSI_HOST_PARAM_HWADDRESS from the
> driver.  Can we just match that against the iBFT's MAC address?  If it


I think this is doable.

A question about hba/non-hba modes.

In non-hba mode in /sys/firmware/ibft/ethernetX/ is there a link to the netdevice. And in hba-mode is there a link too? You have to load iscsi_ibft then load the network driver for the link to appear.
Comment 4 Mike Christie 2011-04-13 18:16:30 UTC 
(In reply to comment #3)
> (In reply to comment #2)
> > Comment from Michael Chan on possible solution
> > 
> > > 1. Modify iscsi_ibft so it checks for BIFT. We can add some sort of callouts or
> > > driver override or something like we normally do, so drivers can hook in
> > > somehow.
> > > 
> > 
> > The search is done in userspace, right?  So send NETLINK to driver?
> 
> I actually just meant that bnx2i would hook into iscsi_ibft/iscsi_ibft_find.c
> and tell it to look for your hba mode sig.

Bah, I guess no matter what we do, we need to support hba mode. And so we need to modify the kernel's iscsi_ibft/iscsi_ibft_find.c code. Then as another bz we need to fix the iscsi tools to do the right thing. Will clone this bz for the kernel part.
Comment 5 Jamal Mir 2011-04-13 20:32:43 UTC 
(In reply to comment #3)
> (In reply to comment #2)
> > Comment from Michael Chan on possible solution
> > 
> > > 1. Modify iscsi_ibft so it checks for BIFT. We can add some sort of callouts or
> > > driver override or something like we normally do, so drivers can hook in
> > > somehow.
> > > 
> > 
> > The search is done in userspace, right?  So send NETLINK to driver?
> I actually just meant that bnx2i would hook into iscsi_ibft/iscsi_ibft_find.c
> and tell it to look for your hba mode sig. Then if found iscsi_boot_sysfs would
> have some new file /sys/firmware/ibft/ethernet/offload that would print "yes"
> in this case.
> iscsi tools would then check that and if set use bnx2i if not use iscsi_tcp.
> > 
> > Alternatively, we already can get the ISCSI_HOST_PARAM_HWADDRESS from the
> > driver.  Can we just match that against the iBFT's MAC address?  If it
> I think this is doable.

> A question about hba/non-hba modes.
> In non-hba mode in /sys/firmware/ibft/ethernetX/ is there a link to the
> netdevice. And in hba-mode is there a link too? You have to load iscsi_ibft
> then load the network driver for the link to appear.

I used the iscsi boot driver ( the one with standard signature) and I was able to verify that there's a link to the netdevice in both HBA and non-HBA mode.

I changed to /sys/firmware/ibft/ethernet0/ folder and executed ls -al command. One of listing was the following:
	 "device -> ../../../devices/pci0000:00/0000:00:1c.0/0000:1c:00.0"
Jamal
Comment 6 Mike Christie 2011-04-14 06:34:10 UTC 
(In reply to comment #3)
> > 
> > Alternatively, we already can get the ISCSI_HOST_PARAM_HWADDRESS from the
> > driver.  Can we just match that against the iBFT's MAC address?  If it
> 
> 
> I think this is doable.

I implemented a patch for this here:
http://groups.google.com/group/open-iscsi/browse_thread/thread/fd276b1a29ca5d13

Will make it into a rpm when I am doing working on the kernel part to check for BIFT, so it can all be tested together.
Comment 7 Michael Chan 2011-04-14 22:19:03 UTC 
Broadcom QA tested above open-iscsi patch with this kernel patch:

http://groups.google.com/group/open-iscsi/browse_thread/thread/14d4d6fda9ce77aa

It seems to work in hba mode and non-hba mode.
Comment 11 Jamal Mir 2011-04-29 17:07:52 UTC 
I got the snapshot 5 yesterday and tried to perform DVD install in two different ways. HBA mode enabled and HBA mode disabled.

1) HBA Mode Enabled - Everything seemed to work just fine. I checked the  
   connection after target discovery and verified that the transport was set to 
   bnx2i. After file copying was done and before reboot, I verified transport 
   was still set to bnx2i. But after reboot and when system finished 
   installation, the transport was set to tcp. I used "iscsiadm -m session" 
   command throughout. Also, I checked the node and iface configuration. They 
   both were configured with bnx2i transport as I expected. It seems that the 
   initrd does not attempt to establish a connection using the bnx2i transport.

2) HBA Mode Disabled - Everything seemed to work well until the system gave 
   this message "kernel panic - not syncing: attempted to kill init!" at the 
   first boot after done copying files.

Jamal
Comment 12 Mike Christie 2011-04-29 18:37:53 UTC 
(In reply to comment #11)
> I got the snapshot 5 yesterday and tried to perform DVD install in two
> different ways. HBA mode enabled and HBA mode disabled.
> 
> 1) HBA Mode Enabled - Everything seemed to work just fine. I checked the  
>    connection after target discovery and verified that the transport was set to 
>    bnx2i. After file copying was done and before reboot, I verified transport 
>    was still set to bnx2i. But after reboot and when system finished 
>    installation, the transport was set to tcp. I used "iscsiadm -m session" 
>    command throughout. Also, I checked the node and iface configuration. They 
>    both were configured with bnx2i transport as I expected. It seems that the 
>    initrd does not attempt to establish a connection using the bnx2i transport.
> 

Could you boot into the system and run the command

// make sure iscsid is not running
#killall iscsid

// run command that initrd should be doing
#iscsistart -b
// that should outpout something about using bnx2i.
// you can also run
iscsiadm -m session

and check if a session got created.

If you are booting from the iscsi_tcp session you can run those commands above with the iscsi_tcp session running. The iscsiadm -m session should just show multiple sessions.


> 2) HBA Mode Disabled - Everything seemed to work well until the system gave 
>    this message "kernel panic - not syncing: attempted to kill init!" at the 
>    first boot after done copying files.

So you saw the iscsi session get created and saw the disk get setup, right?
Comment 13 Mike Christie 2011-04-29 18:38:58 UTC 
(In reply to comment #12)
> (In reply to comment #11)
> > I got the snapshot 5 yesterday and tried to perform DVD install in two
> > different ways. HBA mode enabled and HBA mode disabled.
> > 
> > 1) HBA Mode Enabled - Everything seemed to work just fine. I checked the  
> >    connection after target discovery and verified that the transport was set to 
> >    bnx2i. After file copying was done and before reboot, I verified transport 
> >    was still set to bnx2i. But after reboot and when system finished 
> >    installation, the transport was set to tcp. I used "iscsiadm -m session" 
> >    command throughout. Also, I checked the node and iface configuration. They 
> >    both were configured with bnx2i transport as I expected. It seems that the 
> >    initrd does not attempt to establish a connection using the bnx2i transport.
> > 
> 
> Could you boot into the system and run the command
> 
> // make sure iscsid is not running
> #killall iscsid
> 
> // run command that initrd should be doing
> #iscsistart -b
> // that should outpout something about using bnx2i.
> // you can also run
> iscsiadm -m session
> 
> and check if a session got created.
> 
> If you are booting from the iscsi_tcp session you can run those commands above
> with the iscsi_tcp session running. The iscsiadm -m session should just show
> multiple sessions.
> 

Oh yeah, for this though, make sure that the iscsi init scripts have not created a session using bnx2i before running those commands. If the iscsi init scripts have just log it out first

iscsiadm -m node -T yourtarget -p ip -I the.bnx2i.iface -u
Comment 14 Jamal Mir 2011-04-29 19:20:45 UTC 
(In reply to comment #12)
> (In reply to comment #11)
> > I got the snapshot 5 yesterday and tried to perform DVD install in two
> > different ways. HBA mode enabled and HBA mode disabled.
> > 
> > 1) HBA Mode Enabled - Everything seemed to work just fine. I checked the  
> >    connection after target discovery and verified that the transport was set to 
> >    bnx2i. After file copying was done and before reboot, I verified transport 
> >    was still set to bnx2i. But after reboot and when system finished 
> >    installation, the transport was set to tcp. I used "iscsiadm -m session" 
> >    command throughout. Also, I checked the node and iface configuration. They 
> >    both were configured with bnx2i transport as I expected. It seems that the 
> >    initrd does not attempt to establish a connection using the bnx2i transport.
> > 
> Could you boot into the system and run the command
> // make sure iscsid is not running
> #killall iscsid
> // run command that initrd should be doing
> #iscsistart -b
> // that should outpout something about using bnx2i.
> // you can also run
> iscsiadm -m session
> and check if a session got created.
> If you are booting from the iscsi_tcp session you can run those commands above
> with the iscsi_tcp session running. The iscsiadm -m session should just show
> multiple sessions.

I killed the iscsid process using "killall iscsid" and I did the following:
root@rh6u1ss5-x8664 Desktop]# ps -A | grep iscsi
  110 ?        00:00:00 iscsi_eh
  516 ?        00:00:00 iscsi_q_5
 1315 ?        00:00:00 brcm_iscsiuio
[root@rh6u1ss5-x8664 Desktop]# 
[root@rh6u1ss5-x8664 Desktop]# 
[root@rh6u1ss5-x8664 Desktop]# 
[root@rh6u1ss5-x8664 Desktop]# iscsiadm -m session
tcp: [1] 172.16.100.7:3260,1 iqn.1991-05.com.microsoft:rh6u0.dvdinstall.oob
[root@rh6u1ss5-x8664 Desktop]# 
[root@rh6u1ss5-x8664 Desktop]# 
[root@rh6u1ss5-x8664 Desktop]# 
[root@rh6u1ss5-x8664 Desktop]# iscsistart -b
iscsistart: transport class version 2.0-870. iscsid version 2.0-872
iscsistart: version 2.0-872
iscsistart: Logging into iqn.1991-05.com.microsoft:rh6u0.dvdinstall.oob 172.16.100.7:3260,1
iscsistart: initiator reported error (15 - session exists)
[root@rh6u1ss5-x8664 Desktop]# 

It does not seem that iscsiatart is connecting using bnx2i transport. Also at boot I looked at output of iscsistart and it looks just like the one above. No mention to bnx2i.



> > 2) HBA Mode Disabled - Everything seemed to work well until the system gave 
> >    this message "kernel panic - not syncing: attempted to kill init!" at the 
> >    first boot after done copying files.
> So you saw the iscsi session get created and saw the disk get setup, right?

I did not see any seesion being created nor a disk being setup. the first message was the kernel panic message.
Comment 15 Mike Christie 2011-04-29 19:45:11 UTC 
What is the iscsi-initaitor-utils version?

Do

rpm -q iscsi-initiator-utils

You need at least iscsi-initiator-utils-6.2.0.872-20. I put some rpms here:

http://people.redhat.com/mchristi/iscsi/rhel6.1/iscsi-initiator-utils/

If that does not work can you run with debugging

iscsistart -b -d 8


then send everything that gets spit out.
Comment 16 Jamal Mir 2011-04-29 21:21:42 UTC 
(In reply to comment #15)
> What is the iscsi-initaitor-utils version?
> Do
> rpm -q iscsi-initiator-utils
> You need at least iscsi-initiator-utils-6.2.0.872-20. I put some rpms here:
> http://people.redhat.com/mchristi/iscsi/rhel6.1/iscsi-initiator-utils/

I have iscsi-initiator-utils-6.2.0.872-21.el6.x86_64


> If that does not work can you run with debugging
> iscsistart -b -d 8
> then send everything that gets spit out.

See attachment error.txt for output of iscsistart -b -d 8
Comment 17 Jamal Mir 2011-04-29 21:22:58 UTC 
Created attachment 495867 [details]
output of iscsistart -b -d 8 command

output of iscsistart -b -d 8 command.
Comment 18 Mike Christie 2011-04-29 23:19:19 UTC 
(In reply to comment #17)
> Created attachment 495867 [details]
> output of iscsistart -b -d 8 command
> 
> output of iscsistart -b -d 8 command.

Is that when the card is hba mode or non hba mode?

This message:
iscsistart: Could not match 00:10:18:88:e7:c4 to host
indicates that we could not match that MAC address to a iscsi offload card so we think it is non-hba mode.

If you do

iscsiadm -m host -P 1

do you see the bnx2i offload port with that MAC?
Comment 19 Jamal Mir 2011-04-30 00:15:04 UTC 
(In reply to comment #18)
> (In reply to comment #17)
> > Created attachment 495867 [details]
> > output of iscsistart -b -d 8 command
> > 
> > output of iscsistart -b -d 8 command.
> Is that when the card is hba mode or non hba mode?

NOTE: Please note that I am booting from image created from DVD install and not from local H/D.

It is definitely in HBA Mode as I can not boot to image in non-HBA Mode as I explained previously.

> This message:
> iscsistart: Could not match 00:10:18:88:e7:c4 to host
> indicates that we could not match that MAC address to a iscsi offload card so
> we think it is non-hba mode.

The MAC address I see at boot after enabling HBA Mode is 00:10:18:88:e7:c5 so I know that ibft table has this MAC address.

Is it possible to try this with the NIC we sent you?

> If you do
> iscsiadm -m host -P 1
> do you see the bnx2i offload port with that MAC?

[root@rh6u1ss5-x8664 Desktop]# iscsiadm -m host -P 1
Host Number: 5
	State: running
	Transport: tcp
	Initiatorname: <empty>
	IPaddress: 172.16.100.222
	HWaddress: <empty>
	Netdev: <empty>
Host Number: 7
	State: running
	Transport: bnx2i
	Initiatorname: <empty>
	IPaddress: <empty>
	HWaddress: 00:10:18:88:e7:c7
	Netdev: eth1
Host Number: 8
	State: running
	Transport: bnx2i
	Initiatorname: <empty>
	IPaddress: <empty>
	HWaddress: 00:10:18:88:e7:c5
	Netdev: eth0




[root@rh6u1ss5-x8664 Desktop]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:10:18:88:E7:C4  
          inet addr:172.16.100.222  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::210:18ff:fe88:e7c4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:175406 errors:0 dropped:0 overruns:0 frame:0
          TX packets:93038 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:102467198 (97.7 MiB)  TX bytes:11545652 (11.0 MiB)
          Interrupt:16 Memory:e0000000-e07fffff 

eth1      Link encap:Ethernet  HWaddr 00:10:18:88:E7:C6  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
          Interrupt:17 Memory:e1000000-e17fffff
Comment 20 Mike Christie 2011-04-30 02:20:03 UTC 
(In reply to comment #19)
> Is it possible to try this with the NIC we sent you?
> 

Huh???? Are there 2 Jamal Mirs there :) Michael Chan has asked some Jamal Mir twice to help me get my card working.
Comment 21 Mike Christie 2011-04-30 02:29:37 UTC 
(In reply to comment #19)
> > This message:
> > iscsistart: Could not match 00:10:18:88:e7:c4 to host
> > indicates that we could not match that MAC address to a iscsi offload card so
> > we think it is non-hba mode.
> 
> The MAC address I see at boot after enabling HBA Mode is 00:10:18:88:e7:c5 so 
> know that ibft table has this MAC address.

Where do you see this? In the BIOS screen or in /sys/firmware/ibft/ethernetX/mac? The mac sysfs file prints out the MAC address we found in the ibft info.

The iscsistart log message above prints that out too. If you run

iscsiadm -m fw

do you see the mac that is in the ibft sysfs mac files or something different?
Comment 22 Jamal Mir 2011-04-30 04:07:38 UTC 
(In reply to comment #21)
> (In reply to comment #19)
> > > This message:
> > > iscsistart: Could not match 00:10:18:88:e7:c4 to host
> > > indicates that we could not match that MAC address to a iscsi offload card so
> > > we think it is non-hba mode.
> > 
> > The MAC address I see at boot after enabling HBA Mode is 00:10:18:88:e7:c5 so 
> > know that ibft table has this MAC address.
> Where do you see this? In the BIOS screen or in
> /sys/firmware/ibft/ethernetX/mac? The mac sysfs file prints out the MAC address

The BIOS screen.

> we found in the ibft info.
> The iscsistart log message above prints that out too. If you run
> iscsiadm -m fw
> do you see the mac that is in the ibft sysfs mac files or something different?

I will have to try this on Monday.
Comment 23 Mike Christie 2011-05-03 06:33:38 UTC 
Jamal,

Did you get the answers to the previous questions?

Here, it looks like in hba mode the iscsi tools and sysfs ibft tree show the mac of the iscsi offload mac as expected.

For me though I get different results than you.

In hba mode, boot fails with the kernel panic not syncing error.  Before I get that error, I get a error about not being able to get the boot entry from ibft. Looking into that now.
Comment 24 Mike Christie 2011-05-03 06:46:39 UTC 
(In reply to comment #23)
> In hba mode, boot fails with the kernel panic not syncing error.  Before I get
> that error, I get a error about not being able to get the boot entry from ibft.
> Looking into that now.

This was a user error. ibft data was not getting setup by the broadcom bios stuff.
Comment 25 Mike Christie 2011-05-04 03:19:08 UTC 
Hey Jamal,

I think that non-hba mode works but hba mode does not. I think for hba mode the bnx2i driver is not getting loaded before iscsistart is run.

Can you confirm that you see the same thing?
Comment 26 Jamal Mir 2011-05-04 16:53:28 UTC 
(In reply to comment #25)
> Hey Jamal,
> I think that non-hba mode works but hba mode does not. I think for hba mode the
> bnx2i driver is not getting loaded before iscsistart is run.
> Can you confirm that you see the same thing?


non-hba mode did not work for me. The first phase of installation goes well but system panic at first boot. I am going to repeat the non-hba installation and I'll send screen captures of the whole process.
Comment 27 Jamal Mir 2011-05-04 23:07:34 UTC 
Created attachment 496938 [details]
non-HBA DVD installation steps

Mike C.

I attached screen shots of the steps up to the point of failure.
Comment 28 Mike Christie 2011-05-05 05:06:15 UTC 
For the issue where for offload boot the bnx2i module is not in the initramfs and loaded we have this bz:
https://bugzilla.redhat.com/show_bug.cgi?id=701864
Comment 29 Mike Christie 2011-05-05 05:09:23 UTC 
(In reply to comment #27)
> Created attachment 496938 [details]
> non-HBA DVD installation steps
> 
> Mike C.
> 
> I attached screen shots of the steps up to the point of failure.

It seems to fail because iscsi_ibft is not added in this case.

iscsi: registered transport (tcp)
FATAL: Module ibft not found.

Cloning this for dracut and installer maintainers.
Comment 33 Laura Bailey 2011-05-12 06:31:57 UTC 
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Support for the Broadcom NetXtreme II Network Adapter has been added, allowing users to use iSCSI Tools when booting in both Host Bus Adapter (HBA) mode and non-HBA mode.
Comment 36 Laura Bailey 2011-05-16 01:12:38 UTC 
Deleted Technical Notes Contents.

Old Contents:
Support for the Broadcom NetXtreme II Network Adapter has been added, allowing users to use iSCSI Tools when booting in both Host Bus Adapter (HBA) mode and non-HBA mode.
Comment 37 Gideon Naim 2011-05-16 15:18:32 UTC 
There is still a problem that iSCSI offload installation works but on the next reboot there is still some dracut or anaconda issue. It would be helpful if these teams can help us find a workaround or perhaps a z-stream/maintenance update.
Comment 38 Mike Christie 2011-05-16 22:18:06 UTC 
(In reply to comment #37)
> There is still a problem that iSCSI offload installation works but on the next
> reboot there is still some dracut or anaconda issue. It would be helpful if
> these teams can help us find a workaround or perhaps a z-stream/maintenance
> update.

We were working on that in the dracut bz:
https://bugzilla.redhat.com/show_bug.cgi?id=701864
Right now, it looks like we do not have a work around.

The issue we were trying to fix in the iscsi tools is fixed in this bz.
Comment 39 errata-xmlrpc 2011-05-19 14:14:59 UTC 
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0733.html