RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 835768 - SR-IOV: Given 14 VFs to RHEL7 guest but only 2 enabled
Summary: SR-IOV: Given 14 VFs to RHEL7 guest but only 2 enabled
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: kernel
Version: 7.0
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: Xen Maintainance List
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks: 741684
TreeView+ depends on / blocked
 
Reported: 2012-06-27 04:57 UTC by Wei Shi
Modified: 2012-10-16 14:52 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-07-04 11:59:24 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
rhel7-guest-dmesg (34.04 KB, text/plain)
2012-06-27 04:57 UTC, Wei Shi 		no flags Details
rhel7-guest-messages-log (464.04 KB, text/plain)
2012-06-27 04:58 UTC, Wei Shi 		no flags Details
View All

Description Wei Shi 2012-06-27 04:57:39 UTC 
Created attachment 594673 [details]
rhel7-guest-dmesg

Description of problem:
  Assigned 14 VFs to guest domain, after lunching the domU, xm indicates all VFs is assigned,but in the guest, only 2 nics can be founded by command 'lspci'
  I have tried this case in rhel6.3, no problem, all 14 VFs is detected by guest domain.

  Host machine details:
  Manufacturer: Hewlett-Packard
  Product Name: HP Z400 Workstation
  CPU:          Intel(R) Xeon(R) CPU W3520  @ 2.67GHz   4-threads
  Nic:          Intel 82576 with two ports

  Guest domain:
    rhel7.0-hvm-guest

Version-Release number of selected component (if applicable):
  Host:  RHEL5.8(2.6.18-318.el5xen) Intel(R) Xeon(R) CPU
  Guest: RHEL7.0(RHEL-7.0-20120620.n.0) HVM x86_64(no i386 is out, untested)

How reproducible:
100%

Steps to Reproduce:
  On a AMD machine with Intel 82576 with two ports. Parameter "iommu=no-intremap" and "pci_pt_e820_access=on" have already added.

1.[host]lspci | grep 82576
have a Physical 82576 NIC with two ports

2.[host]Reload igb
# modprobe -r igb
# modprobe igb max_vfs=7

3.[host]Load pciback driver:
# modprobe pciback
check whether pciback was loaded:
# lsmod | grep pciback

4.[host]Get device ID of VFs:
# lspci -D | grep "82576 Virtual Function"

5.[host]Unbind PCI device from native driver and Bind PCI device to
pciback driver.

6.[host]Make sure the VFs has been hidden from Dom0 already:
# xm pci-list-assignable-devices
0000:03:10.1
0000:03:10.0
0000:03:10.2
0000:03:10.3
0000:03:10.4
0000:03:10.5
0000:03:10.6
0000:03:10.7
0000:03:11.0
0000:03:11.1
0000:03:11.2
0000:03:11.3
0000:03:11.4
0000:03:11.5

# xm pci-list [$domU]
VSlt domain   bus   slot   func
6      0    3     10     0      
7      0    3     10     1      
0      0    3     10     2      
0      0    3     10     3      
0      0    3     10     4      
0      0    3     10     5      
0      0    3     10     6      
0      0    3     10     7      
0      0    3     11     0      
0      0    3     11     1      
0      0    3     11     2      
0      0    3     11     3      
0      0    3     11     4      
0      0    3     11     5

7.[host]Create the guest with 14 VF devices' ID as parameter
add the following item to guest cfg file:
pci=['0000:03:10.0','0000:03:10.1','0000:03:10.2','0000:03:10.3','0000:03:10.4','0000:03:10.5','0000:03:10.6','0000:03:10.7','0000:03:11.0','0000:03:11.1','0000:03:11.2','0000:03:11.3','0000:03:11.4','0000:03:11.5']

lunch rhel7 guest domain

8.[host]Check all VFs is assigned to guest
#xm pci-list-assignable-devices
nothing

9.[guest]Check nics in guest by command lspci, only two VF are visible:
# lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:01.3 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:06.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:07.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:11.0 SCSI storage controller: XenSource, Inc. Xen Platform Device (rev 01)
  
Actual results:
  Only two nics are visible in the rhel 7 guest

Expected results:
  All VF network interfaces of guest should be visible and configured correctly

Additional info:
  Some additional guest log file is attached as attachments.
  This situation not occur on rhel6.3-hvm-guest.x86_64
Comment 1 Wei Shi 2012-06-27 04:58:59 UTC 
Created attachment 594675 [details]
rhel7-guest-messages-log
Comment 2 Laszlo Ersek 2012-06-28 19:13:26 UTC 
Can you please provide an lspci from within the working RHEL-6.3 HVM guest? Thank you.
Comment 3 Wei Shi 2012-06-29 02:38:20 UTC 
(In reply to comment #2)
> Can you please provide an lspci from within the working RHEL-6.3 HVM guest?
> Thank you.

The following output is result of lspci command in rhel6.3-hvm-x86_64 guest:

[root@dhcp-8-202 ~]# lspci 
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:01.3 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:04.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:05.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:06.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:07.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:08.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:09.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:0a.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:0b.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:0c.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:0d.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:0e.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:0f.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:11.0 SCSI storage controller: XenSource, Inc. Xen Platform Device (rev 01)
00:12.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 20)
Comment 4 Laszlo Ersek 2012-06-29 08:40:38 UTC 
Thank you. Interesting. RHEL-5 userspace, "tools/ioemu/vl.h" says:

  /* PCI slot 6~7 support ACPI PCI hot plug */
  #define PHP_SLOT_START  (6)
  #define PHP_SLOT_END    (8)
  #define PHP_SLOT_LEN    (PHP_SLOT_END - PHP_SLOT_START)
  #define PHP_TO_PCI_SLOT(x) (x + PHP_SLOT_START)
  #define PCI_TO_PHP_SLOT(x) (x - PHP_SLOT_START)
  #define PHP_DEVFN_START (PHP_SLOT_START << 3)
  #define PHP_DEVFN_END   (PHP_SLOT_END << 3)

and both of the following (RHEL-7 guest) quotes from comment 0 seem to be consistent with that:

> # xm pci-list [$domU]
> VSlt domain   bus   slot   func
> 6      0    3     10     0      
> 7      0    3     10     1      
> 0      0    3     10     2      
> 0      0    3     10     3      
> [... VSlt=0 in the rest (VSlt is "Virtual Slot") ...]

and

> 9.[guest]Check nics in guest by command lspci, only two VF are visible:
> # lspci
> [...]
> 00:06.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev
> 01)
> 00:07.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev
> 01)

I'm actually surprised that the RHEL-6 guest is not affected by this restriction.

Can you please boot with
- "loglvl=all guest_loglvl=all" on the xen.gz command line,
- "ignore_loglevel" on the dom0 vmlinuz command line,
- and "ignore_loglevel earlyprintk=xenboot" on the RHEL-6 and RHEL-7 guest command lines?

I'd like to compare the PCI initialization in the guests.
Comment 5 Laszlo Ersek 2012-06-29 08:43:55 UTC 
Hi Don,

can you please take a quick look? Could this be related to xen_emul_unplug in RHEL-6?

Thank you,
Laszlo
Comment 6 Wei Shi 2012-06-29 09:45:39 UTC 
(In reply to comment #4)
> Thank you. Interesting. RHEL-5 userspace, "tools/ioemu/vl.h" says:
> 
>   /* PCI slot 6~7 support ACPI PCI hot plug */
>   #define PHP_SLOT_START  (6)
>   #define PHP_SLOT_END    (8)
>   #define PHP_SLOT_LEN    (PHP_SLOT_END - PHP_SLOT_START)
>   #define PHP_TO_PCI_SLOT(x) (x + PHP_SLOT_START)
>   #define PCI_TO_PHP_SLOT(x) (x - PHP_SLOT_START)
>   #define PHP_DEVFN_START (PHP_SLOT_START << 3)
>   #define PHP_DEVFN_END   (PHP_SLOT_END << 3)
> 
> and both of the following (RHEL-7 guest) quotes from comment 0 seem to be
> consistent with that:
> 
> > # xm pci-list [$domU]
> > VSlt domain   bus   slot   func
> > 6      0    3     10     0      
> > 7      0    3     10     1      
> > 0      0    3     10     2      
> > 0      0    3     10     3      
> > [... VSlt=0 in the rest (VSlt is "Virtual Slot") ...]
> 
> and
> 
> > 9.[guest]Check nics in guest by command lspci, only two VF are visible:
> > # lspci
> > [...]
> > 00:06.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev
> > 01)
> > 00:07.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev
> > 01)
> 
> I'm actually surprised that the RHEL-6 guest is not affected by this
> restriction.
> 
> Can you please boot with
> - "loglvl=all guest_loglvl=all" on the xen.gz command line,
> - "ignore_loglevel" on the dom0 vmlinuz command line,
> - and "ignore_loglevel earlyprintk=xenboot" on the RHEL-6 and RHEL-7 guest
> command lines?
> 
> I'd like to compare the PCI initialization in the guests.

After i added "ignore_loglevel earlyprintk=xenboot" to kernel cmdline, both rhel6 and rhel7 guest can't be booted up.
Comment 7 Wei Shi 2012-06-29 09:49:53 UTC 
(In reply to comment #5)
> Hi Don,
> 
> can you please take a quick look? Could this be related to xen_emul_unplug
> in RHEL-6?
> 
> Thank you,
> Laszlo

Yes, it's relate to this parameter, after i adjust both rhel6 and rhel7 to the same value, the lspci resualt is just the same.

PS:
  before i change the value, the xen_emul_unplug parameter's values is:
  rhel6.3--- ide-disks
  rhel7.0--- ide-all
  they set different values.
Comment 8 Laszlo Ersek 2012-06-29 10:05:01 UTC 
So, if you set (non-default) "ide-disks" for rhel7.0, so that it matches rhel6.3, then rhel7.0 works as well?

Similarly, if you set (non-default) "ide-all" for rhel6.3, so that it matches rhel7.0, then rhel6.3 stops working as well? (Meaning, it only sees two VFs?)

Thanks.
Comment 9 Wei Shi 2012-06-29 10:11:09 UTC 
(In reply to comment #8)
> So, if you set (non-default) "ide-disks" for rhel7.0, so that it matches
> rhel6.3, then rhel7.0 works as well?
> 
> Similarly, if you set (non-default) "ide-all" for rhel6.3, so that it
> matches rhel7.0, then rhel6.3 stops working as well? (Meaning, it only sees
> two VFs?)
> 
> Thanks.

Yes, exactly as you say.
Comment 10 Don Dutile (Red Hat) 2012-06-29 14:57:15 UTC 
(In reply to comment #5)
> Hi Don,
> 
> can you please take a quick look? Could this be related to xen_emul_unplug
> in RHEL-6?
> 
> Thank you,
> Laszlo

Well, I see you got your answer(s) in c#7 & c#9.
does the xen_emul_unplug mess up ACPI with too many slots not being hot-pluggable??? ... I'm curious...
Comment 11 Don Dutile (Red Hat) 2012-06-29 15:01:57 UTC 
(In reply to comment #5)
> Hi Don,
> 
> can you please take a quick look? Could this be related to xen_emul_unplug
> in RHEL-6?
> 
> Thank you,
> Laszlo

I just recalled -- there was a bug in the (upstream) xen emul code that would disable VFs inadvertently when 'all' was set.  did that patch not get into rhel5's xen code?  but, 2 VFs make it .... hmmm.... I wonder if the bug exists for all slots after last xen-blk device (0.11.0).
Comment 14 Laszlo Ersek 2012-07-04 11:59:24 UTC 
* I checked the guest configs, they seem correct. The only differences are:
name, disk backing file, and vif MAC.

* (Hints: [1] [2]) In the RHEL-7 guest, I added the following line to
/etc/default/grub:

  GRUB_TERMINAL="serial console"

then ran

  grub2-mkconfig -o /boot/grub2/grub.cfg

which allowed me to edit the kernel command line in xm console / grub also
in the RHEL-7 guest.

* RHEL-6 guest with no xen_emul_unplug:

  [root@dhcp-8-202 ~]# cat /proc/cmdline
    ro root=/dev/mapper/vg_dhcp8202-lv_root rd_NO_LUKS LANG=en_US.UTF-8
    rd_NO_MD rd_LVM_LV=vg_dhcp8202/lv_root SYSFONT=latarcyrheb-sun16
    KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rd_LVM_LV=vg_dhcp8202/lv_swap
    console=tty0 console=ttyS0,115200n8

  [root@dhcp-8-202 ~]# uname -r
    2.6.32-279.el6.x86_64

  [root@dhcp-8-202 ~]# lspci
    [...]
    00:06.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:07.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    [...]

* RHEL-6 guest with xen_emul_unplug=ide-disks:

  [root@dhcp-8-202 ~]# cat /proc/cmdline
    ro root=/dev/mapper/vg_dhcp8202-lv_root rd_NO_LUKS LANG=en_US.UTF-8
    rd_NO_MD rd_LVM_LV=vg_dhcp8202/lv_root SYSFONT=latarcyrheb-sun16
    KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rd_LVM_LV=vg_dhcp8202/lv_swap
    xen_emul_unplug=ide-disks console=tty0 console=ttyS0,115200n8

  [root@dhcp-8-202 ~]# uname -r
    2.6.32-279.el6.x86_64

  [root@dhcp-8-202 ~]# lspci
    [...]
    00:03.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:04.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:05.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:06.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:07.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:08.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:09.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:0a.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:0b.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:0c.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:0d.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:0e.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:0f.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    [...]

* RHEL-7 guest with no xen_emul_unplug:

  [root@rhel7 ~]# cat /proc/cmdline
    BOOT_IMAGE=/vmlinuz-3.3.0-0.15.el7.x86_64
    root=/dev/mapper/vg_rhel7-lv_root ro rd.md=0 rd.dm=0 SYSFONT=True
    rd.lvm.lv=vg_rhel7/lv_root KEYTABLE=us rd.luks=0
    rd.lvm.lv=vg_rhel7/lv_swap LANG=en_US.UTF-8 console=tty0
    console=ttyS0,112500

  [root@rhel7 ~]# uname -r
    3.3.0-0.15.el7.x86_64

  [root@rhel7 ~]# lspci
    [...]
    00:06.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:07.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    [...]

* RHEL-7 guest with xen_emul_unplug=ide-disks:

  [root@rhel7 ~]# cat /proc/cmdline
    BOOT_IMAGE=/vmlinuz-3.3.0-0.15.el7.x86_64
    root=/dev/mapper/vg_rhel7-lv_root ro rd.md=0 rd.dm=0 SYSFONT=True
    rd.lvm.lv=vg_rhel7/lv_root KEYTABLE=us rd.luks=0
    rd.lvm.lv=vg_rhel7/lv_swap LANG=en_US.UTF-8 xen_emul_unplug=ide-disks
    console=tty0 console=ttyS0,112500

  [root@rhel7 ~]# uname -r
    3.3.0-0.15.el7.x86_64

  [root@rhel7 ~]# lspci
    [...]
    00:03.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:04.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:05.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:06.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:07.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:08.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:09.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:0a.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:0b.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:0c.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:0d.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:0e.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:0f.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    [...]

* "ide-all" is an invalid value for xen_emul_unplug, both in RHEL-6 and
   RHEL-7.

* Since the RHEL-6 and the RHEL-7 kernels behave identically with the same
xen_emul_unplug setting, and their behaviors match in the default case as
well (ie. with "xen_emul_unplug" absent), I'm closing this as NOTABUG. The
apparently different behavior is caused by whatever composes the kernel
command lines differently.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=801650#c61
[2] http://post-office.corp.redhat.com/archives/tech-list/2011-December/msg00080.html
Note You need to log in before you can comment on or make changes to this bug.