Bug 602451 - [Emulex 6.0 feat] Include SRIOV support to be2net
Summary: [Emulex 6.0 feat] Include SRIOV support to be2net
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kernel
Version: 6.0
Hardware: All
OS: Linux
high
high
Target Milestone: rc
: 6.0
Assignee: Ivan Vecera
QA Contact: Red Hat Kernel QE team
URL:
Whiteboard:
Depends On:
Blocks: 604729 617187 738269
TreeView+ depends on / blocked
 
Reported: 2010-06-09 20:53 UTC by laurie barry
Modified: 2014-01-31 13:36 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
The SR-IOV functionality of the Emulex be2net driver on the BE3 asic is considered a Technology Preview in Red Hat Enterprise Linux 6. Please contact your Emulex account representative to obtain the minimum UCNA firmware revision level required for SRIOV support.
Clone Of:
: 617187 (view as bug list)
Environment:
Last Closed: 2010-11-15 14:07:10 UTC
Target Upstream Version:


Attachments (Terms of Use)
ADD PCI SRIOV support (14.72 KB, patch)
2010-06-14 14:51 UTC, Subbu Seetharaman
no flags Details | Diff
Patch to add SRIOV support to be2net (13.30 KB, patch)
2010-06-14 15:40 UTC, Subbu Seetharaman
no flags Details | Diff
dmesg output as requested by subbu (5.48 KB, application/octet-stream)
2010-06-18 13:56 UTC, Ram Pai
no flags Details
lspci output (3.93 KB, application/octet-stream)
2010-06-18 13:56 UTC, Ram Pai
no flags Details
iproute package that should support VF (332.66 KB, application/octet-stream)
2010-06-23 09:39 UTC, Ivan Vecera
ivecera: review? (subbu.seetharaman)
Details
Patch to eliminate two confusing error messages with MAC address setting for VF (1.80 KB, patch)
2010-06-24 14:23 UTC, Subbu Seetharaman
no flags Details | Diff
Patch to change the logic to identify VF (1.50 KB, patch)
2010-07-07 14:27 UTC, Subbu Seetharaman
no flags Details | Diff

Description laurie barry 2010-06-09 20:53:00 UTC
Per discussions with Andrius; we are requesting the SRIOV Tech Preview support be added to the be2net driver for RHEL6.

RH has been reviewing the kernel dependencies and working to determine what is feasible at this stage of the release.

Laurie Barry

Comment 2 RHEL Product and Program Management 2010-06-09 21:13:13 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.

Comment 3 Andrius Benokraitis 2010-06-09 21:15:54 UTC
Can you please reference the code/bugzilla that has the SRIOV support requested in RHEL 6?

I believe I see it already proposed in RHEL 5.6 in the following:
https://bugzilla.redhat.com/show_bug.cgi?id=568388#c22

Comment 6 Subbu Seetharaman 2010-06-14 14:51:43 UTC
Created attachment 423860 [details]
ADD PCI SRIOV support

Patch to add SRIOV support to be2net.

Comment 7 Andrius Benokraitis 2010-06-14 15:32:55 UTC
Ivan - was this the patch you had?

Comment 8 Subbu Seetharaman 2010-06-14 15:40:23 UTC
Created attachment 423876 [details]
Patch to add SRIOV support to be2net

New patch to add SRIOV support to be2net.  Includes following two upstream pacthes 

ba343c7736b36d62d276e20383588bcf9403d6c6
        be2net: Adding PCI SRIOV support
84e5b9f75b48fe4a1e4ee72698230701439d0805
        be2net: Patch removes redundant while statement in loop.

Comment 9 Ivan Vecera 2010-06-14 15:55:26 UTC
Yes, I already grabbed these patches from upstream (net-next).

Comment 14 Andrius Benokraitis 2010-06-15 16:54:20 UTC
Emulex/IBM - please test this ASAP. I will be providing you an email with test kernels since we do not have SRIOV hardware in house.

Comment 15 Ram Pai 2010-06-15 19:07:49 UTC
Andrius, 

rpm -i kernel-2.6.32-33.el6ivtest.1.x86_64.rpm kernel-firmware-2.6.32-33.el6.noarch.rpm
error: Failed dependencies:
	kernel-firmware >= 2.6.32-33.el6ivtest.1 is needed by kernel-2.6.32-33.el6ivtest.1.x86_64

 I will force install hoping that the dependency string is improperly captured.

FYI
RP

Comment 16 Ram Pai 2010-06-15 21:14:23 UTC
Emulex/ServerEngine,

  Do I need any newer firmware. I get the following error when I enable VFs

be2net 0000:24:00.1: PCI INT B disabled
be2net 0000:24:00.0: PCI INT A disabled
be2net 0000:24:00.0: PCI INT A -> GSI 32 (level, low) -> IRQ 32
be2net 0000:24:00.0: setting latency timer to 64
be2net 0000:24:00.0: irq 57 for MSI/MSI-X
be2net 0000:24:00.0: irq 58 for MSI/MSI-X
32bit 0000:24:04.0 uses non-identity mapping
be2net 0000:24:00.0: Emulex OneConnect 10Gbps NIC port 1
be2net 0000:24:00.1: PCI INT B -> GSI 42 (level, low) -> IRQ 42
be2net 0000:24:00.1: setting latency timer to 64
be2net 0000:24:00.1: irq 102 for MSI/MSI-X
be2net 0000:24:00.1: irq 103 for MSI/MSI-X
eth3: Link up
be2net 0000:24:00.1: Emulex OneConnect 10Gbps NIC port 0
be2net 0000:24:04.0: enabling device (0040 -> 0042)
be2net 0000:24:04.0: setting latency timer to 64
64bit 0000:24:04.0 uses identity mapping
be2net 0000:24:04.0: Error in cmd completion - opcode 1, compl 1, extd 1
be2net 0000:24:04.0: Error in cmd completion - opcode 13, compl 5, extd 22
ADDRCONF(NETDEV_UP): eth5: link is not ready
be2net 0000:24:04.0: Emulex OneConnect 10Gbps NIC initialization failed
be2net 0000:24:08.0: enabling device (0040 -> 0042)
be2net 0000:24:08.0: setting latency timer to 64
64bit 0000:24:08.0 uses identity mapping
be2net 0000:24:08.0: Error in cmd completion - opcode 1, compl 1, extd 1
be2net 0000:24:08.0: Error in cmd completion - opcode 13, compl 5, extd 22
be2net 0000:24:08.0: Emulex OneConnect 10Gbps NIC initialization failed


Let me know.
RP

Comment 17 Ram Pai 2010-06-15 22:46:25 UTC
well even 2.6.35-rc3 has the same issue. This very much looks like a firmware thingy.. 

RP

Comment 18 Subbu Seetharaman 2010-06-16 15:17:53 UTC
You need newer firmware as well as card with new personality in the EEPROM.

Comment 19 Ram Pai 2010-06-16 16:32:52 UTC
Subbu,

    Can you arrange for the new firmware/card sent my way?

thanks,
RP

Comment 20 Subbu Seetharaman 2010-06-16 18:19:11 UTC
Ram,

The card IBM has must have the right personality and you need only f/w upgrade.  We see some problems that f/w engineers are debugging. As soon as we have a fix, we will send you the new f/w.

Thanks.
Subbu

Comment 21 Subbu Seetharaman 2010-06-17 13:28:09 UTC
Installation of kernel-2.6.32-33.el6ivtest.1.x86_64.rpm fails with dependency problem.  Force installing using --nodeps throws some other errors and I am unable to boot the new kernel.  Any help ?

[root@sriov-rhel6 tmp]# rpm -Uvh kernel-2.6.32-33.el6ivtest.1.x86_64.rpm
error: Failed dependencies:
        kernel-firmware >= 2.6.32-33.el6ivtest.1 is needed by kernel-2.6.32-33.el6ivtest.1.x86_64
[root@sriov-rhel6 tmp]#
[root@sriov-rhel6 tmp]# rpm -Uvh kernel-2.6.32-33.el6ivtest.1.x86_64.rpm --nodeps
Preparing...                ########################################### [100%]
   1:kernel                 ########################################### [100%]
grubby fatal error: unable to find a suitable template
grubby: doing this would leave no kernel entries. Not writing out new config.
grubby fatal error: unable to find a suitable template

Comment 22 Andrius Benokraitis 2010-06-17 13:34:18 UTC
Technical note added. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.

New Contents:
SR-IOV functionality in the Emulex be2net driver will be Tech Preview in RHEL 6.0.

Comment 23 Ram Pai 2010-06-17 15:00:27 UTC
Subbu,
  
      Does it hang while you boot the new kernel, or you are unable to find an entry in grub to boot the new kernel.

       If it is the former, try installing the firmware rpm available at the same site. Hopefully that should help.
 
       If it is the later, create grub entries corresponding to the new kernel.
        
 
RP

Comment 24 Subbu Seetharaman 2010-06-18 12:19:34 UTC
The gub entry was not made by the rpm becasue  of the error it reported.  So, I manually added it and on reboot, grub showed the boot menu; but when I picked the -33 kernel it failed with the error "File not found". 

If this kernel comes up for you, can you please send me the full dmesg 
and the output of "lspci -v -d19a2:*" after insmod be2net.ko num_vfs=1 ?

Thanks.

Subbu

Comment 25 Ram Pai 2010-06-18 13:56:05 UTC
Created attachment 425126 [details]
dmesg output  as requested by subbu

Comment 26 Ram Pai 2010-06-18 13:56:59 UTC
Created attachment 425127 [details]
lspci output

Comment 27 Ram Pai 2010-06-18 16:11:39 UTC
Ok. I received the be3 based controllers. However the OS fails to allocate MMIO resource to its SRIOV pcie capability due to a conflict with one other resource.

Here is the message in dmesg logs

pnp 00:0a: mem resource (0xffc00000-0xffffffff) overlaps 0000:24:00.0 BAR 7 (0xffffc000-0x10003bfff), disabling
pnp 00:0a: mem resource (0xffc00000-0xffffffff) overlaps 0000:24:00.1 BAR 7 (0xffffc000-0x10003bfff), disabling

I am investigating
RP

Comment 28 Ryan Lerch 2010-06-21 02:12:35 UTC
Technical note updated. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.

Diffed Contents:
@@ -1 +1 @@
-SR-IOV functionality in the Emulex be2net driver will be Tech Preview in RHEL 6.0.+The SR-IOV functionality of the Emulex be2net driver is considered a Technology Preview in Red Hat Enterprise Linux 6.

Comment 29 Subbu Seetharaman 2010-06-21 14:40:37 UTC
Ivan / Andrius,

Because of the problem I mentioned in the comment #21, we are unable to test it in the kernel you pointed to.  We tested this in Beta Snapshot 6.  The driver loads and initializes without any problems; but the ip command to configure the MAC address for VFs is failing :

[root@sriov-rhel6 ip]# ip link set eth0 vf 0 mac 00:16:88:AA:BB:CC
Error: either "dev" is duplicate, or "vf" is a garbage.

We tried with ip command  built from tip of git tree, we get the following error :

[root@sriov-rhel6 ip]# ./ip link set eth0 vf 0 mac 00:16:88:AA:BB:CC
RTNETLINK answers: Operation not permitted

Does snapshot6 include all the netlink and iproute patches required  for setting VF MAC address ?

Comment 30 Subbu Seetharaman 2010-06-21 14:43:52 UTC
Ram,

Dont bother testing SRIOV in BE2.  Pleae dont plan to support SRIOV in BE2.  Please use BE3 for all your SRIOV testing.

Subbu

Comment 31 laurie barry 2010-06-21 15:27:04 UTC
Technical note updated. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.

Diffed Contents:
@@ -1 +1,3 @@
-The SR-IOV functionality of the Emulex be2net driver is considered a Technology Preview in Red Hat Enterprise Linux 6.+The SR-IOV functionality of the Emulex be2net driver on the BE3 asic is considered a Technology Preview in Red Hat Enterprise Linux 6. 
+
+Please contact your Emulex account representative to obtain the minimum UCNA firmware revision level required for SRIOV support.

Comment 32 Ivan Vecera 2010-06-21 15:57:18 UTC
(In reply to comment #29)
> Ivan / Andrius,
> 
> Because of the problem I mentioned in the comment #21, we are unable to test it
> in the kernel you pointed to.  We tested this in Beta Snapshot 6.  The driver
> loads and initializes without any problems; but the ip command to configure the
> MAC address for VFs is failing :
> 
> [root@sriov-rhel6 ip]# ip link set eth0 vf 0 mac 00:16:88:AA:BB:CC
> Error: either "dev" is duplicate, or "vf" is a garbage.
> 
> We tried with ip command  built from tip of git tree, we get the following
> error :
> 
> [root@sriov-rhel6 ip]# ./ip link set eth0 vf 0 mac 00:16:88:AA:BB:CC
> RTNETLINK answers: Operation not permitted
> 
Hmm, EPERM... Subbu, it seems that this code is returned from be_set_vf_mac function in be_main.c file. Is possile that SRIOV was not initialized properly thus 'adapter->sriov_enabled' is zero?

Comment 33 Ram Pai 2010-06-21 16:42:12 UTC
RedHat,

General question:  How can a customer know if a driver is supported or in Tech Preview. Is there some command that one can run on the driver to figure that out?


RP

Comment 34 Andrius Benokraitis 2010-06-21 17:21:27 UTC
(In reply to comment #33)
> RedHat,
> 
> General question:  How can a customer know if a driver is supported or in Tech
> Preview. Is there some command that one can run on the driver to figure that
> out?
> 
> 
> RP    

Release Notes only. BTW - in this case the driver isn't Tech Preview, the SRIOV functionality is. Hence why it needs to be in the release notes so it can provide additional clarification if needed.

Comment 35 Ram Pai 2010-06-21 22:10:14 UTC
Subbu,

        Ok. Since I did not have the sources to 2.6.32-33.el6ivtest kernel, I 
a) took the snap4 kernel, 
b) applied the MMIO resource allocation backported patches submitted in        
         https://bugzilla.redhat.com/show_bug.cgi?id=587729. 
      My platform needs these patches unfortunately.
c) Applied the be2net sriov patches attached in this bugzilla.
   I had to comment out a function entry from be_netdev_ops
        /* .ndo_set_vf_mac               = be_set_vf_mac */
   since snap4 kernel does not support that field yet.


With the above changes MMIO resource allocation failure disappears. 
NOTE: I am using a brand new be3 controller.

However I still see the following:

be2net 0000:24:04.2: enabling device (0040 -> 0042)
be2net 0000:24:04.2: setting latency timer to 64
  alloc irq_desc for 91 on node -1
  alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
be2net 0000:24:04.2: irq 91 for MSI/MSI-X
  alloc irq_desc for 92 on node -1
  alloc kstat_irqs on node -1
alloc irq_2_iommu on node -1
be2net 0000:24:04.2: irq 92 for MSI/MSI-X
be2net 0000:24:04.2: Error in cmd completion - opcode 1, compl 1, extd 1


Do I continue to have the right firmware?

Comment 37 Ivan Vecera 2010-06-22 08:34:59 UTC
(In reply to comment #29)
> Ivan / Andrius,
> ...
> [root@sriov-rhel6 ip]# ip link set eth0 vf 0 mac 00:16:88:AA:BB:CC
> Error: either "dev" is duplicate, or "vf" is a garbage.
> 
Btw. the current iproute does not support link parameters showing/setting for virtual functions. I filled a new bug #606680 for this issue.

Comment 38 Subbu Seetharaman 2010-06-22 12:02:24 UTC
Ivan,

You are right in commeent #32.  SRIOV was enabled correctly; but we were invoking the ipcommand with wrong argumnet. In the setup in which we trid this, eth0 was the interface from a VF and not PF.  Hence adapter->sriov_enadled was zero.  With the right argument (and new iproute command), SRIOV works as expected in the snapshot 6.  We found two other minor driver issues for which we will submit a pacth shortly.  Thanks.

Ram,

You have the right f/w.  You can ignore this mesage for now.  We will fix it in the patch.  If you have the updated iproute utility, you must be able to use it and complete your testing (or you can wait for the patch that will fix this and another wrong error message).

Subbu

Comment 39 Andrius Benokraitis 2010-06-22 19:41:14 UTC
(In reply to comment #35)
> Subbu,
> 
>         Ok. Since I did not have the sources to 2.6.32-33.el6ivtest kernel,

Ram - the source can be found in the same private link location that John Jarvis sent you via email.

Comment 40 Subbu Seetharaman 2010-06-23 06:11:20 UTC
Also, SRIOV seems to work fine in snapshot 6.  I dont think you need 2.6.32-33 kernel.  You will need to use the iproute from the git or the one you got from SE recently.

Comment 41 Ivan Vecera 2010-06-23 09:39:29 UTC
Created attachment 426208 [details]
iproute package that should support VF

Subbu, could you please confirm that the following proposed iproute package works with VFs correctly?

Comment 42 laurie barry 2010-06-23 13:59:24 UTC
Ivan - I have BE3 SRIOV capable UCNA boards ready to drop off to RH in Westford, MA on my way home this evening.  Who should I direct the boards to in Westford?

Laurie

Comment 43 Peter Martuccelli 2010-06-23 14:09:28 UTC
Drop them off at the front desk and I will pick them up Thursday morning.

Comment 44 laurie barry 2010-06-23 14:28:42 UTC
Will do; I'll put them to your attention.  Thanks Peter.

Laurie

Comment 45 Ram Pai 2010-06-23 19:38:12 UTC

With the new iproute2 package I get the following error

#ip link set eth3 vf 0 mac 00:16:88:AA:BB:CC
RTNETLINK answers: Invalid argument

however 
ifconfig eth3  hw ether 00:16:88:AA:BB:CC
works successfully. Is there a difference between the two?

anyway going forward, I associated an ip address to the VF. And tried to ping a remote ip. ping fails. tcpdump seems to indicate that the other end is responding to who-is message, but this end is not getting back the message. 

So I associated an ip address with the corresponding PF. ping started working.
I suppose this is by design.

A netperf run passed both ways.
 
However I see wierd behavior like 
a) ip address of the VF getting reset when the PF is configured with an ip address
b) ARP reply message associating mac address of the VF to the ip address of the PF.


In general the behavior looks flaky.

Anyway I will next associate the VF to a guest and see how it behaves. Stay tuned...

BTW: for completeness, my kernel is snap4 based, with platform enablement SRIOV patches, as well the be2net driver patch attached in this bugzilla. I can go with Andreas mentioned kernel ie 2.6.32-33.el6ivtest, if insisted.

Comment 46 Ram Pai 2010-06-23 22:18:18 UTC
passthru'd the VF to a rhel6 guest running 2.6.32-33.el6ivtest kernel version.
VF is seen and the be2net driver loads and grab the VF successfully. However the link does not come on. So no communication to the outside world through the VF. :(

RP

Comment 47 Subbu Seetharaman 2010-06-24 11:16:52 UTC
Ivan,

> Subbu, could you please confirm that the following proposed iproute package
> works with VFs correctly?    

Works fine.

Comment 48 Subbu Seetharaman 2010-06-24 11:49:24 UTC
Ram,

> #ip link set eth3 vf 0 mac 00:16:88:AA:BB:CC
> RTNETLINK answers: Invalid argument

I hope eth3 is the interface corresponding to the PF.  If it is,  I think you need to upgrade to snap6 or 2.6.32-33.el6ivtest.

> ifconfig eth3  hw ether 00:16:88:AA:BB:CC
> works successfully. Is there a difference between the two?

If eth3 is the i/f corresponding to PF, this step is wrong.  This MAC address setting is needed only for the i/f corresponding to VF. The PF gets the MAC address in the EEPROM and there is no need to assign new.

> Is there a difference between the two?

Yes. MAC address setting through both iproute as well as ifconfig should succeed before you can successfully ping.  iproute configuration programs the MAC address in the NIC and ifconfig in the netdev structure in host.

> VF is seen and the be2net driver loads and grab the VF successfully. However
> the link does not come on. So no communication to the outside world through
> the VF. :(

What hypervisor are you using ?  We have tested this in XEN and not seen this problem. If you are connecting the network cable to only one port, please make sure that the passthru'ed VF corresponds to the port on which the link  is up.  Since the PF<->VF association is not very obvious, I frequently make this mistake.  In any case until you get the MAC address assignment done using iproute, ping will not work.

Subbu

Comment 49 Ram Pai 2010-06-24 13:53:07 UTC
>>#ip link set eth3 vf 0 mac 00:16:88:AA:BB:CC
>> RTNETLINK answers: Invalid argument

> I hope eth3 is the interface corresponding to the PF.  If it is,  I think you
> need to upgrade to snap6 or 2.6.32-33.el6ivtest.

eth3 is the interface for the VF.  The PF already has a mac address.

>> ifconfig eth3  hw ether 00:16:88:AA:BB:CC
>> works successfully. Is there a difference between the two?

>If eth3 is the i/f corresponding to PF, this step is wrong.  This MAC address
>setting is needed only for the i/f corresponding to VF. The PF gets the MAC
>address in the EEPROM and there is no need to assign new.

right. as mentioned above eth3 is VF

>> Is there a difference between the two?

>Yes. MAC address setting through both iproute as well as ifconfig should
>succeed before you can successfully ping.  iproute configuration programs the
>MAC address in the NIC and ifconfig in the netdev structure in host.

Ok. ip command failed, but ifconfig passed.


>> VF is seen and the be2net driver loads and grab the VF successfully. However
>> the link does not come on. So no communication to the outside world through
>> the VF. :(

> What hypervisor are you using ?  We have tested this in XEN and not seen this
> problem. If you are connecting the network cable to only one port, please make
> sure that the passthru'ed VF corresponds to the port on which the link  is up. 
> Since the PF<->VF association is not very obvious, I frequently make this
> mistake.  In any case until you get the MAC address assignment done using
> iproute, ping will not work.

I am using KVM hypervisor. Yes i did check for the association of the PF and the VF, Infact the same VF had the link on when used on the host, but failed to show link-up when passthru'd to the guest.

Comment 50 Subbu Seetharaman 2010-06-24 14:23:33 UTC
Created attachment 426591 [details]
Patch to eliminate two confusing error messages with MAC address setting for VF


Path to :

1. Fix wrong failure check to log error after set MAC address command
2. Avoid call to delete MAC address on VF if none set.

Will be submitted to netdev shortly.

Subbu

Comment 51 Ram Pai 2010-06-24 21:22:09 UTC
Downloaded 2.6.32-33.el6ivtest sources, applied my platform enable sriov patches, applied subbu's patch in comment 50, built and booted the kernel.

ensured i that the latest iproute package in comment 41 is installed.

rmmod be2net
modprobe be2net num_vfs=1

module loaded correctly. No more 'Error in cmd completion' kind of messages. Good!

Noticed that the two PFs are eth7 and eth8. The corresponding VFs interfaces are eth3 and eth4. Both these interfaces have there mac address set to 00:00:00:00:00:00.

btw, eth7 has its link up.

now tried the following

#ip link set eth3 vf 0 mac 00:16:88:AA:BB:CC
RTNETLINK answers: Operation not permitted


After some investigation looks the culprit is this code in 

---------------------------------------------------------------------------
static int be_set_vf_mac(struct net_device *netdev, int vf, u8 *mac)
{
        struct be_adapter *adapter = netdev_priv(netdev);
        int status;

        if (!adapter->sriov_enabled)
                return -EPERM;
---------------------------------------------------------------------------

note: the net_device received by this function is that of the virtual function. Checking for sriov_enabled on that function will always return false.
right?

I changed the code to extract out the be_adapter structure of the physical function.  I added the following hack before the check.

adapter = (struct be_adapter *)container_of(adapter->pdev->physfn, struct be_adapter, pdev);

it made progress.. but.. now it is stuck in

Call Trace:
 [<ffffffff814da6e6>] ? _spin_lock_bh+0x16/0x40
 [<ffffffffa0245e8f>] ? be_cmd_pmac_del+0x3f/0xb0 [be2net]
 [<ffffffffa0241380>] ? be_set_vf_mac+0x120/0x190 [be2net]
 [<ffffffff8141facb>] ? do_setlink+0x58b/0x830
 [<ffffffff81120411>] ? lru_cache_add_lru+0x21/0x40
 [<ffffffff8126929f>] ? nla_parse+0xef/0x110
 [<ffffffff814201be>] ? rtnl_newlink+0x44e/0x530
 [<ffffffff8141f320>] ? rtnetlink_rcv_msg+0x1e0/0x220
 [<ffffffff8141f140>] ? rtnetlink_rcv_msg+0x0/0x220
 [<ffffffff814366b9>] ? netlink_rcv_skb+0xa9/0xd0
 [<ffffffff8141f125>] ? rtnetlink_rcv+0x25/0x40
 [<ffffffff8143631e>] ? netlink_unicast+0x2de/0x2f0
 [<ffffffff81436cb0>] ? netlink_sendmsg+0x200/0x2e0
 [<ffffffff814023de>] ? sock_sendmsg+0x11e/0x150
 [<ffffffff81090ba0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8110bd9e>] ? find_get_page+0x1e/0xa0
 [<ffffffff8110d95e>] ? filemap_fault+0xbe/0x510
 [<ffffffff81400ac4>] ? move_addr_to_kernel+0x64/0x70
 [<ffffffff8140c109>] ? verify_iovec+0x69/0xc0
 [<ffffffff814026a3>] ? sys_sendmsg+0x233/0x3a0
 [<ffffffff8114db47>] ? alloc_pages_current+0x87/0xd0
 [<ffffffff81049da7>] ? pte_alloc_one+0x37/0x50
 [<ffffffff81133dfd>] ? handle_mm_fault+0x1ed/0x2b0
 [<ffffffff810d38d2>] ? audit_syscall_entry+0x252/0x280
 [<ffffffff81013172>] ? system_call_fastpath+0x16/0x1b

Am i entirely on the wrong path?

Comment 52 Subbu Seetharaman 2010-06-25 06:57:35 UTC
Ram,

#ip link set eth3 vf 0 mac 00:16:88:AA:BB:CC
RTNETLINK answers: Operation not permitted

If eth3 is the VF this is wrong.  You should  be specifying the PF interface here.  The command  should be :

# ip link set eth7 vf 0 mac 00:16:88:AA:BB:CC

What we are doing here is settig the MAC address for the first VF hanging of the PF interface  eth7 (which in your case is eth3. 

Hope this  clears the confusion.

Thanks.

Subbu

Comment 53 Ram Pai 2010-06-25 14:51:33 UTC
OOps. did I get it entirely wrong???? And you have been saying it all along. Ok. will try with the new gained knowledge and report back.

RP

Comment 54 Ram Pai 2010-06-25 18:13:44 UTC
Does not work for me.

#ip link set eth7 vf 0 mac 00:16:88:AA:BB:CC

dont' see the mac address getting set on the eth3 interface. Cannot associate an ip address with the interface. Well I am lost, it just does not work.

Subbu, can you send me the set of instructions on how to make this adapter working?

Comment 55 Subbu Seetharaman 2010-06-28 10:16:37 UTC
Ram,

The ip command sets the MAC address for the VF in the NIC.  Looks like that step has been successful.  However that is not enough for the stack to know the MAC address.  You must also run the ifconfig command to set the MAC address.  I think this is the step you were missing.  

Assuming that eth3 and eth4 are the VF interfaces corresponding to the PF i/f eth7, these are the commands that you must run to configure the MAC address for the VFs.

# ip link set eth7 vf 0 mac 00:16:88:AA:BB:AA
# ip link set eth7 vf 1 mac 00:16:88:AA:BB:AB
# ifconfig eth3 hw ether 00:16:88:AA:BB:AA
# ifconfig eth4 hw ether 00:16:88:AA:BB:AB


After this, you must be able to assign IP address for eth3 and eth4 and ping out.  Let us get things working upto this point before assigning the VFs to  VMs.

Subbu

Comment 56 Ram Pai 2010-06-30 15:20:50 UTC
Ok. that works. I can ping the other end and vice-versa after associating an mac address using both ip and the ifconfig command. However I am not sure why both are needed and just one is not sufficient. 

I moved on to the next step of assigning the VFs to the VMs.  Turns out the be2net driver in the VM thinks that it is operating on a PF. The culprit there
is 

#define be_physfn(adapter) (!adapter->pdev->is_virtfn)

is this logic correct? not being VF does not mean it is a PF. Does it?  
Should'nt this check be  (adpater->pdev->is_physfn)    ?

note: in the VM, sriov_init() on that device returns without doing anything. The result is is_virtfn not set, leaving it at zero.

Comment 57 Subbu Seetharaman 2010-07-01 06:45:16 UTC
> However I am not sure why both are needed and just one is not sufficient. 

As  I said before, the iproute programs the h/w.  The guestOS stack also needs to know the MAC address and that is done through the ifconfig. In a regular NIC configuration, both of these happen in the same adminitrative domain and hence the ifconfig command can manage both.  In SRIOV, for security reasons, we do not want to allow a user with admin privilege in a VM to change the MAC address in the NIC.  I agree it is a bit confusing. Hopefully better mechanisms will evolve.

> I moved on to the next step of assigning the VFs to the VMs.  Turns out the
> be2net driver in the VM thinks that it is operating on a PF.

The driver runing the VM should do a few things differently and we belive we have the right things for this in RH6. A recent patch in RH 5.6 also fixes this.  What GuesOS and driver are you using in the VM ?

Subbu

Comment 58 Ram Pai 2010-07-01 15:03:39 UTC
>The driver runing the VM should do a few things differently and we belive we
>have the right things for this in RH6. A recent patch in RH 5.6 also fixes
>this.  What GuesOS and driver are you using in the VM ?

Both my host and guest are RHEL6, running the same version of the kernel provided by Andrius. And I am using KVM.

RP

Comment 59 Aristeu Rozanski 2010-07-01 16:13:30 UTC
Patch(es) available on kernel-2.6.32-42.el6

Comment 61 Ram Pai 2010-07-01 21:05:12 UTC
Aristeu,
  I understand kernel-2.6.32-42.el6 is the snap7 kernel. Any chance I can get a test version now?

RP

Comment 63 Subbu Seetharaman 2010-07-02 14:38:20 UTC

> #define be_physfn(adapter) (!adapter->pdev->is_virtfn)
> 
> is this logic correct? not being VF does not mean it is a PF. Does it?  
> Should'nt this check be  (adpater->pdev->is_physfn)    ?


This logic is broken.  Both is_physfn and is_virtfn are false in the passthrough'ed VF.  Checking (adpater->pdev->is_physfn) will not work since in platforms where SRIOV is not enabled, is_physfn will be 0.  We need another method to distinguish VF from PF.  I have mailed you a driver.  Please try it and if it works, I will attach a patch.


Subbu

Comment 64 Ram Pai 2010-07-02 19:55:02 UTC
Ok. your new patch makes progress. I see that you are now querying the adapter to figure out if it is operating on a PF or a VF. 
However the driver still fails to load in the VM. 

be2net 0000:00:05.0: Error in cmd completion - opcode 50, compl 3, extd 75
be2net 0000:00:05.0: Emulex OneConnect 10Gbps NIC (be3) initialization failed

BTW: you can attach the patch. it makes some progress.
RP

Comment 65 Subbu Seetharaman 2010-07-07 14:27:30 UTC
Created attachment 430082 [details]
Patch to change the logic to identify VF

Patch to change the logic to distinguish VF from PF.  Fixes the problem reported by Ram Pai @ IBM.

Comment 66 Andrius Benokraitis 2010-07-08 04:08:56 UTC
(In reply to comment #65)
> Created an attachment (id=430082) [details]
> Patch to change the logic to identify VF
> 
> Patch to change the logic to distinguish VF from PF.  Fixes the problem
> reported by Ram Pai @ IBM.    

This probably needs to be in a new BZ since this bug is already ON_QA. Will leave that to Ivan though.

Comment 67 Ivan Vecera 2010-07-19 14:29:47 UTC
Subbu, it should be better to open a new BZ...

Comment 68 Subbu Seetharaman 2010-07-22 12:51:51 UTC
Comment on attachment 430082 [details]
Patch to change the logic to identify VF

This fix will be part of the patch in bug 617187

Comment 69 Ivan Vecera 2010-07-22 13:30:49 UTC
Subbu, unfortunately the patch (Patch to eliminate two confusing error messages with MAC address setting for VF) was not applied with this bug report. But nevermind I'm going to apply it with the new bug 617187.
But I can not find related upstream commits... were they submitted to netdev already?

Comment 70 Subbu Seetharaman 2010-07-29 16:28:29 UTC
Comment on attachment 426591 [details]
Patch to eliminate two confusing error messages with MAC address setting for VF

Ivan,

The patch I attached to BZ 617817 covers the necessary changes from this patch.

Comment 71 Subbu Seetharaman 2010-07-29 16:35:50 UTC
Sorry I meant BZ 617187 and not 617817.

Comment 72 Andrius Benokraitis 2010-09-13 18:45:53 UTC
*IMPORTANT*

Partners - please test this bugzilla *immediately* and report back with a comment. Delays in test findings could impact future proposed inclusions due to failed partner test commitments. Your results are very important to Red Hat and its mutual customers.

Comment 73 Subbu Seetharaman 2010-09-15 09:33:32 UTC
Verified on RH6 snapshot 12.

Subbu

Comment 74 Petr Beňas 2010-10-13 11:39:00 UTC
Code review.
Verified the patch is actually being applied in 2.6.32-42.el6 source.
kernel.spec
3448 Patch3063: netdrv-be2net-Add-PCI-SR-IOV-support.patch
6624 ApplyPatch netdrv-be2net-Add-PCI-SR-IOV-support.patch

Comment 75 releng-rhel@redhat.com 2010-11-15 14:07:10 UTC
Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.