Comment from Michael Chan on possible solution

> 1. Modify iscsi_ibft so it checks for BIFT. We can add some sort of callouts or
> driver override or something like we normally do, so drivers can hook in
> somehow.
> 

The search is done in userspace, right?  So send NETLINK to driver?

Alternatively, we already can get the ISCSI_HOST_PARAM_HWADDRESS from the
driver.  Can we just match that against the iBFT's MAC address?  If it matches,
we do offload.  If it doesn't match, we use iscsi_tcp.  We still need userspace
to add a search for the BIFT signature.

(In reply to comment #2)
> Comment from Michael Chan on possible solution
> 
> > 1. Modify iscsi_ibft so it checks for BIFT. We can add some sort of callouts or
> > driver override or something like we normally do, so drivers can hook in
> > somehow.
> > 
> 
> The search is done in userspace, right?  So send NETLINK to driver?

I actually just meant that bnx2i would hook into iscsi_ibft/iscsi_ibft_find.c and tell it to look for your hba mode sig. Then if found iscsi_boot_sysfs would have some new file /sys/firmware/ibft/ethernet/offload that would print "yes" in this case.

iscsi tools would then check that and if set use bnx2i if not use iscsi_tcp.

> 
> Alternatively, we already can get the ISCSI_HOST_PARAM_HWADDRESS from the
> driver.  Can we just match that against the iBFT's MAC address?  If it


I think this is doable.

A question about hba/non-hba modes.

In non-hba mode in /sys/firmware/ibft/ethernetX/ is there a link to the netdevice. And in hba-mode is there a link too? You have to load iscsi_ibft then load the network driver for the link to appear.

(In reply to comment #3)
> (In reply to comment #2)
> > Comment from Michael Chan on possible solution
> > 
> > > 1. Modify iscsi_ibft so it checks for BIFT. We can add some sort of callouts or
> > > driver override or something like we normally do, so drivers can hook in
> > > somehow.
> > > 
> > 
> > The search is done in userspace, right?  So send NETLINK to driver?
> 
> I actually just meant that bnx2i would hook into iscsi_ibft/iscsi_ibft_find.c
> and tell it to look for your hba mode sig.

Bah, I guess no matter what we do, we need to support hba mode. And so we need to modify the kernel's iscsi_ibft/iscsi_ibft_find.c code. Then as another bz we need to fix the iscsi tools to do the right thing. Will clone this bz for the kernel part.

(In reply to comment #3)
> (In reply to comment #2)
> > Comment from Michael Chan on possible solution
> > 
> > > 1. Modify iscsi_ibft so it checks for BIFT. We can add some sort of callouts or
> > > driver override or something like we normally do, so drivers can hook in
> > > somehow.
> > > 
> > 
> > The search is done in userspace, right?  So send NETLINK to driver?
> I actually just meant that bnx2i would hook into iscsi_ibft/iscsi_ibft_find.c
> and tell it to look for your hba mode sig. Then if found iscsi_boot_sysfs would
> have some new file /sys/firmware/ibft/ethernet/offload that would print "yes"
> in this case.
> iscsi tools would then check that and if set use bnx2i if not use iscsi_tcp.
> > 
> > Alternatively, we already can get the ISCSI_HOST_PARAM_HWADDRESS from the
> > driver.  Can we just match that against the iBFT's MAC address?  If it
> I think this is doable.

> A question about hba/non-hba modes.
> In non-hba mode in /sys/firmware/ibft/ethernetX/ is there a link to the
> netdevice. And in hba-mode is there a link too? You have to load iscsi_ibft
> then load the network driver for the link to appear.

I used the iscsi boot driver ( the one with standard signature) and I was able to verify that there's a link to the netdevice in both HBA and non-HBA mode.

I changed to /sys/firmware/ibft/ethernet0/ folder and executed ls -al command. One of listing was the following:
	 "device -> ../../../devices/pci0000:00/0000:00:1c.0/0000:1c:00.0"
Jamal

(In reply to comment #3)
> > 
> > Alternatively, we already can get the ISCSI_HOST_PARAM_HWADDRESS from the
> > driver.  Can we just match that against the iBFT's MAC address?  If it
> 
> 
> I think this is doable.

I implemented a patch for this here:
http://groups.google.com/group/open-iscsi/browse_thread/thread/fd276b1a29ca5d13

Will make it into a rpm when I am doing working on the kernel part to check for BIFT, so it can all be tested together.

Broadcom QA tested above open-iscsi patch with this kernel patch:

http://groups.google.com/group/open-iscsi/browse_thread/thread/14d4d6fda9ce77aa

It seems to work in hba mode and non-hba mode.

I got the snapshot 5 yesterday and tried to perform DVD install in two different ways. HBA mode enabled and HBA mode disabled.

1) HBA Mode Enabled - Everything seemed to work just fine. I checked the  
   connection after target discovery and verified that the transport was set to 
   bnx2i. After file copying was done and before reboot, I verified transport 
   was still set to bnx2i. But after reboot and when system finished 
   installation, the transport was set to tcp. I used "iscsiadm -m session" 
   command throughout. Also, I checked the node and iface configuration. They 
   both were configured with bnx2i transport as I expected. It seems that the 
   initrd does not attempt to establish a connection using the bnx2i transport.

2) HBA Mode Disabled - Everything seemed to work well until the system gave 
   this message "kernel panic - not syncing: attempted to kill init!" at the 
   first boot after done copying files.

Jamal

(In reply to comment #11)
> I got the snapshot 5 yesterday and tried to perform DVD install in two
> different ways. HBA mode enabled and HBA mode disabled.
> 
> 1) HBA Mode Enabled - Everything seemed to work just fine. I checked the  
>    connection after target discovery and verified that the transport was set to 
>    bnx2i. After file copying was done and before reboot, I verified transport 
>    was still set to bnx2i. But after reboot and when system finished 
>    installation, the transport was set to tcp. I used "iscsiadm -m session" 
>    command throughout. Also, I checked the node and iface configuration. They 
>    both were configured with bnx2i transport as I expected. It seems that the 
>    initrd does not attempt to establish a connection using the bnx2i transport.
> 

Could you boot into the system and run the command

// make sure iscsid is not running
#killall iscsid

// run command that initrd should be doing
#iscsistart -b
// that should outpout something about using bnx2i.
// you can also run
iscsiadm -m session

and check if a session got created.

If you are booting from the iscsi_tcp session you can run those commands above with the iscsi_tcp session running. The iscsiadm -m session should just show multiple sessions.


> 2) HBA Mode Disabled - Everything seemed to work well until the system gave 
>    this message "kernel panic - not syncing: attempted to kill init!" at the 
>    first boot after done copying files.

So you saw the iscsi session get created and saw the disk get setup, right?

(In reply to comment #12)
> (In reply to comment #11)
> > I got the snapshot 5 yesterday and tried to perform DVD install in two
> > different ways. HBA mode enabled and HBA mode disabled.
> > 
> > 1) HBA Mode Enabled - Everything seemed to work just fine. I checked the  
> >    connection after target discovery and verified that the transport was set to 
> >    bnx2i. After file copying was done and before reboot, I verified transport 
> >    was still set to bnx2i. But after reboot and when system finished 
> >    installation, the transport was set to tcp. I used "iscsiadm -m session" 
> >    command throughout. Also, I checked the node and iface configuration. They 
> >    both were configured with bnx2i transport as I expected. It seems that the 
> >    initrd does not attempt to establish a connection using the bnx2i transport.
> > 
> 
> Could you boot into the system and run the command
> 
> // make sure iscsid is not running
> #killall iscsid
> 
> // run command that initrd should be doing
> #iscsistart -b
> // that should outpout something about using bnx2i.
> // you can also run
> iscsiadm -m session
> 
> and check if a session got created.
> 
> If you are booting from the iscsi_tcp session you can run those commands above
> with the iscsi_tcp session running. The iscsiadm -m session should just show
> multiple sessions.
> 

Oh yeah, for this though, make sure that the iscsi init scripts have not created a session using bnx2i before running those commands. If the iscsi init scripts have just log it out first

iscsiadm -m node -T yourtarget -p ip -I the.bnx2i.iface -u

(In reply to comment #12)
> (In reply to comment #11)
> > I got the snapshot 5 yesterday and tried to perform DVD install in two
> > different ways. HBA mode enabled and HBA mode disabled.
> > 
> > 1) HBA Mode Enabled - Everything seemed to work just fine. I checked the  
> >    connection after target discovery and verified that the transport was set to 
> >    bnx2i. After file copying was done and before reboot, I verified transport 
> >    was still set to bnx2i. But after reboot and when system finished 
> >    installation, the transport was set to tcp. I used "iscsiadm -m session" 
> >    command throughout. Also, I checked the node and iface configuration. They 
> >    both were configured with bnx2i transport as I expected. It seems that the 
> >    initrd does not attempt to establish a connection using the bnx2i transport.
> > 
> Could you boot into the system and run the command
> // make sure iscsid is not running
> #killall iscsid
> // run command that initrd should be doing
> #iscsistart -b
> // that should outpout something about using bnx2i.
> // you can also run
> iscsiadm -m session
> and check if a session got created.
> If you are booting from the iscsi_tcp session you can run those commands above
> with the iscsi_tcp session running. The iscsiadm -m session should just show
> multiple sessions.

I killed the iscsid process using "killall iscsid" and I did the following:
root@rh6u1ss5-x8664 Desktop]# ps -A | grep iscsi
  110 ?        00:00:00 iscsi_eh
  516 ?        00:00:00 iscsi_q_5
 1315 ?        00:00:00 brcm_iscsiuio
[root@rh6u1ss5-x8664 Desktop]# 
[root@rh6u1ss5-x8664 Desktop]# 
[root@rh6u1ss5-x8664 Desktop]# 
[root@rh6u1ss5-x8664 Desktop]# iscsiadm -m session
tcp: [1] 172.16.100.7:3260,1 iqn.1991-05.com.microsoft:rh6u0.dvdinstall.oob
[root@rh6u1ss5-x8664 Desktop]# 
[root@rh6u1ss5-x8664 Desktop]# 
[root@rh6u1ss5-x8664 Desktop]# 
[root@rh6u1ss5-x8664 Desktop]# iscsistart -b
iscsistart: transport class version 2.0-870. iscsid version 2.0-872
iscsistart: version 2.0-872
iscsistart: Logging into iqn.1991-05.com.microsoft:rh6u0.dvdinstall.oob 172.16.100.7:3260,1
iscsistart: initiator reported error (15 - session exists)
[root@rh6u1ss5-x8664 Desktop]# 

It does not seem that iscsiatart is connecting using bnx2i transport. Also at boot I looked at output of iscsistart and it looks just like the one above. No mention to bnx2i.



> > 2) HBA Mode Disabled - Everything seemed to work well until the system gave 
> >    this message "kernel panic - not syncing: attempted to kill init!" at the 
> >    first boot after done copying files.
> So you saw the iscsi session get created and saw the disk get setup, right?

I did not see any seesion being created nor a disk being setup. the first message was the kernel panic message.

What is the iscsi-initaitor-utils version?

Do

rpm -q iscsi-initiator-utils

You need at least iscsi-initiator-utils-6.2.0.872-20. I put some rpms here:

http://people.redhat.com/mchristi/iscsi/rhel6.1/iscsi-initiator-utils/

If that does not work can you run with debugging

iscsistart -b -d 8


then send everything that gets spit out.

(In reply to comment #15)
> What is the iscsi-initaitor-utils version?
> Do
> rpm -q iscsi-initiator-utils
> You need at least iscsi-initiator-utils-6.2.0.872-20. I put some rpms here:
> http://people.redhat.com/mchristi/iscsi/rhel6.1/iscsi-initiator-utils/

I have iscsi-initiator-utils-6.2.0.872-21.el6.x86_64


> If that does not work can you run with debugging
> iscsistart -b -d 8
> then send everything that gets spit out.

See attachment error.txt for output of iscsistart -b -d 8

Created attachment 495867 [details]
output of iscsistart -b -d 8 command

output of iscsistart -b -d 8 command.

(In reply to comment #17)
> Created attachment 495867 [details]
> output of iscsistart -b -d 8 command
> 
> output of iscsistart -b -d 8 command.

Is that when the card is hba mode or non hba mode?

This message:
iscsistart: Could not match 00:10:18:88:e7:c4 to host
indicates that we could not match that MAC address to a iscsi offload card so we think it is non-hba mode.

If you do

iscsiadm -m host -P 1

do you see the bnx2i offload port with that MAC?

(In reply to comment #18)
> (In reply to comment #17)
> > Created attachment 495867 [details]
> > output of iscsistart -b -d 8 command
> > 
> > output of iscsistart -b -d 8 command.
> Is that when the card is hba mode or non hba mode?

NOTE: Please note that I am booting from image created from DVD install and not from local H/D.

It is definitely in HBA Mode as I can not boot to image in non-HBA Mode as I explained previously.

> This message:
> iscsistart: Could not match 00:10:18:88:e7:c4 to host
> indicates that we could not match that MAC address to a iscsi offload card so
> we think it is non-hba mode.

The MAC address I see at boot after enabling HBA Mode is 00:10:18:88:e7:c5 so I know that ibft table has this MAC address.

Is it possible to try this with the NIC we sent you?

> If you do
> iscsiadm -m host -P 1
> do you see the bnx2i offload port with that MAC?

[root@rh6u1ss5-x8664 Desktop]# iscsiadm -m host -P 1
Host Number: 5
	State: running
	Transport: tcp
	Initiatorname: <empty>
	IPaddress: 172.16.100.222
	HWaddress: <empty>
	Netdev: <empty>
Host Number: 7
	State: running
	Transport: bnx2i
	Initiatorname: <empty>
	IPaddress: <empty>
	HWaddress: 00:10:18:88:e7:c7
	Netdev: eth1
Host Number: 8
	State: running
	Transport: bnx2i
	Initiatorname: <empty>
	IPaddress: <empty>
	HWaddress: 00:10:18:88:e7:c5
	Netdev: eth0




[root@rh6u1ss5-x8664 Desktop]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:10:18:88:E7:C4  
          inet addr:172.16.100.222  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::210:18ff:fe88:e7c4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:175406 errors:0 dropped:0 overruns:0 frame:0
          TX packets:93038 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:102467198 (97.7 MiB)  TX bytes:11545652 (11.0 MiB)
          Interrupt:16 Memory:e0000000-e07fffff 

eth1      Link encap:Ethernet  HWaddr 00:10:18:88:E7:C6  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
          Interrupt:17 Memory:e1000000-e17fffff

(In reply to comment #19)
> Is it possible to try this with the NIC we sent you?
> 

Huh???? Are there 2 Jamal Mirs there :) Michael Chan has asked some Jamal Mir twice to help me get my card working.

(In reply to comment #19)
> > This message:
> > iscsistart: Could not match 00:10:18:88:e7:c4 to host
> > indicates that we could not match that MAC address to a iscsi offload card so
> > we think it is non-hba mode.
> 
> The MAC address I see at boot after enabling HBA Mode is 00:10:18:88:e7:c5 so 
> know that ibft table has this MAC address.

Where do you see this? In the BIOS screen or in /sys/firmware/ibft/ethernetX/mac? The mac sysfs file prints out the MAC address we found in the ibft info.

The iscsistart log message above prints that out too. If you run

iscsiadm -m fw

do you see the mac that is in the ibft sysfs mac files or something different?

(In reply to comment #21)
> (In reply to comment #19)
> > > This message:
> > > iscsistart: Could not match 00:10:18:88:e7:c4 to host
> > > indicates that we could not match that MAC address to a iscsi offload card so
> > > we think it is non-hba mode.
> > 
> > The MAC address I see at boot after enabling HBA Mode is 00:10:18:88:e7:c5 so 
> > know that ibft table has this MAC address.
> Where do you see this? In the BIOS screen or in
> /sys/firmware/ibft/ethernetX/mac? The mac sysfs file prints out the MAC address

The BIOS screen.

> we found in the ibft info.
> The iscsistart log message above prints that out too. If you run
> iscsiadm -m fw
> do you see the mac that is in the ibft sysfs mac files or something different?

I will have to try this on Monday.

Jamal,

Did you get the answers to the previous questions?

Here, it looks like in hba mode the iscsi tools and sysfs ibft tree show the mac of the iscsi offload mac as expected.

For me though I get different results than you.

In hba mode, boot fails with the kernel panic not syncing error.  Before I get that error, I get a error about not being able to get the boot entry from ibft. Looking into that now.

(In reply to comment #23)
> In hba mode, boot fails with the kernel panic not syncing error.  Before I get
> that error, I get a error about not being able to get the boot entry from ibft.
> Looking into that now.

This was a user error. ibft data was not getting setup by the broadcom bios stuff.

Hey Jamal,

I think that non-hba mode works but hba mode does not. I think for hba mode the bnx2i driver is not getting loaded before iscsistart is run.

Can you confirm that you see the same thing?

(In reply to comment #25)
> Hey Jamal,
> I think that non-hba mode works but hba mode does not. I think for hba mode the
> bnx2i driver is not getting loaded before iscsistart is run.
> Can you confirm that you see the same thing?


non-hba mode did not work for me. The first phase of installation goes well but system panic at first boot. I am going to repeat the non-hba installation and I'll send screen captures of the whole process.

Created attachment 496938 [details]
non-HBA DVD installation steps

Mike C.

I attached screen shots of the steps up to the point of failure.

For the issue where for offload boot the bnx2i module is not in the initramfs and loaded we have this bz:
https://bugzilla.redhat.com/show_bug.cgi?id=701864

(In reply to comment #27)
> Created attachment 496938 [details]
> non-HBA DVD installation steps
> 
> Mike C.
> 
> I attached screen shots of the steps up to the point of failure.

It seems to fail because iscsi_ibft is not added in this case.

iscsi: registered transport (tcp)
FATAL: Module ibft not found.

Cloning this for dracut and installer maintainers.

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Support for the Broadcom NetXtreme II Network Adapter has been added, allowing users to use iSCSI Tools when booting in both Host Bus Adapter (HBA) mode and non-HBA mode.

Deleted Technical Notes Contents.

Old Contents:
Support for the Broadcom NetXtreme II Network Adapter has been added, allowing users to use iSCSI Tools when booting in both Host Bus Adapter (HBA) mode and non-HBA mode.

There is still a problem that iSCSI offload installation works but on the next reboot there is still some dracut or anaconda issue. It would be helpful if these teams can help us find a workaround or perhaps a z-stream/maintenance update.

(In reply to comment #37)
> There is still a problem that iSCSI offload installation works but on the next
> reboot there is still some dracut or anaconda issue. It would be helpful if
> these teams can help us find a workaround or perhaps a z-stream/maintenance
> update.

We were working on that in the dracut bz:
https://bugzilla.redhat.com/show_bug.cgi?id=701864
Right now, it looks like we do not have a work around.

The issue we were trying to fix in the iscsi tools is fixed in this bz.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0733.html