Bug 835768

Summary: SR-IOV: Given 14 VFs to RHEL7 guest but only 2 enabled
Product: Red Hat Enterprise Linux 7 Reporter: Wei Shi <wshi>
Component: kernelAssignee: Xen Maintainance List <xen-maint>
Status: CLOSED NOTABUG QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.0CC: ddutile, drjones, leiwang, lersek, pasik, qguan, qwan, xen-maint, yuzhou
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-07-04 11:59:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 741684    
Attachments:
Description Flags
rhel7-guest-dmesg
none
rhel7-guest-messages-log none

Description Wei Shi 2012-06-27 04:57:39 UTC 
Created attachment 594673 [details]
rhel7-guest-dmesg

Description of problem:
  Assigned 14 VFs to guest domain, after lunching the domU, xm indicates all VFs is assigned,but in the guest, only 2 nics can be founded by command 'lspci'
  I have tried this case in rhel6.3, no problem, all 14 VFs is detected by guest domain.

  Host machine details:
  Manufacturer: Hewlett-Packard
  Product Name: HP Z400 Workstation
  CPU:          Intel(R) Xeon(R) CPU W3520  @ 2.67GHz   4-threads
  Nic:          Intel 82576 with two ports

  Guest domain:
    rhel7.0-hvm-guest

Version-Release number of selected component (if applicable):
  Host:  RHEL5.8(2.6.18-318.el5xen) Intel(R) Xeon(R) CPU
  Guest: RHEL7.0(RHEL-7.0-20120620.n.0) HVM x86_64(no i386 is out, untested)

How reproducible:
100%

Steps to Reproduce:
  On a AMD machine with Intel 82576 with two ports. Parameter "iommu=no-intremap" and "pci_pt_e820_access=on" have already added.

1.[host]lspci | grep 82576
have a Physical 82576 NIC with two ports

2.[host]Reload igb
# modprobe -r igb
# modprobe igb max_vfs=7

3.[host]Load pciback driver:
# modprobe pciback
check whether pciback was loaded:
# lsmod | grep pciback

4.[host]Get device ID of VFs:
# lspci -D | grep "82576 Virtual Function"

5.[host]Unbind PCI device from native driver and Bind PCI device to
pciback driver.

6.[host]Make sure the VFs has been hidden from Dom0 already:
# xm pci-list-assignable-devices
0000:03:10.1
0000:03:10.0
0000:03:10.2
0000:03:10.3
0000:03:10.4
0000:03:10.5
0000:03:10.6
0000:03:10.7
0000:03:11.0
0000:03:11.1
0000:03:11.2
0000:03:11.3
0000:03:11.4
0000:03:11.5

# xm pci-list [$domU]
VSlt domain   bus   slot   func
6      0    3     10     0      
7      0    3     10     1      
0      0    3     10     2      
0      0    3     10     3      
0      0    3     10     4      
0      0    3     10     5      
0      0    3     10     6      
0      0    3     10     7      
0      0    3     11     0      
0      0    3     11     1      
0      0    3     11     2      
0      0    3     11     3      
0      0    3     11     4      
0      0    3     11     5

7.[host]Create the guest with 14 VF devices' ID as parameter
add the following item to guest cfg file:
pci=['0000:03:10.0','0000:03:10.1','0000:03:10.2','0000:03:10.3','0000:03:10.4','0000:03:10.5','0000:03:10.6','0000:03:10.7','0000:03:11.0','0000:03:11.1','0000:03:11.2','0000:03:11.3','0000:03:11.4','0000:03:11.5']

lunch rhel7 guest domain

8.[host]Check all VFs is assigned to guest
#xm pci-list-assignable-devices
nothing

9.[guest]Check nics in guest by command lspci, only two VF are visible:
# lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:01.3 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:06.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:07.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:11.0 SCSI storage controller: XenSource, Inc. Xen Platform Device (rev 01)
  
Actual results:
  Only two nics are visible in the rhel 7 guest

Expected results:
  All VF network interfaces of guest should be visible and configured correctly

Additional info:
  Some additional guest log file is attached as attachments.
  This situation not occur on rhel6.3-hvm-guest.x86_64
Comment 1 Wei Shi 2012-06-27 04:58:59 UTC 
Created attachment 594675 [details]
rhel7-guest-messages-log
Comment 2 Laszlo Ersek 2012-06-28 19:13:26 UTC 
Can you please provide an lspci from within the working RHEL-6.3 HVM guest? Thank you.
Comment 3 Wei Shi 2012-06-29 02:38:20 UTC 
(In reply to comment #2)
> Can you please provide an lspci from within the working RHEL-6.3 HVM guest?
> Thank you.

The following output is result of lspci command in rhel6.3-hvm-x86_64 guest:

[root@dhcp-8-202 ~]# lspci 
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:01.3 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:04.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:05.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:06.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:07.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:08.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:09.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:0a.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:0b.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:0c.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:0d.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:0e.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:0f.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
00:11.0 SCSI storage controller: XenSource, Inc. Xen Platform Device (rev 01)
00:12.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 20)
Comment 4 Laszlo Ersek 2012-06-29 08:40:38 UTC 
Thank you. Interesting. RHEL-5 userspace, "tools/ioemu/vl.h" says:

  /* PCI slot 6~7 support ACPI PCI hot plug */
  #define PHP_SLOT_START  (6)
  #define PHP_SLOT_END    (8)
  #define PHP_SLOT_LEN    (PHP_SLOT_END - PHP_SLOT_START)
  #define PHP_TO_PCI_SLOT(x) (x + PHP_SLOT_START)
  #define PCI_TO_PHP_SLOT(x) (x - PHP_SLOT_START)
  #define PHP_DEVFN_START (PHP_SLOT_START << 3)
  #define PHP_DEVFN_END   (PHP_SLOT_END << 3)

and both of the following (RHEL-7 guest) quotes from comment 0 seem to be consistent with that:

> # xm pci-list [$domU]
> VSlt domain   bus   slot   func
> 6      0    3     10     0      
> 7      0    3     10     1      
> 0      0    3     10     2      
> 0      0    3     10     3      
> [... VSlt=0 in the rest (VSlt is "Virtual Slot") ...]

and

> 9.[guest]Check nics in guest by command lspci, only two VF are visible:
> # lspci
> [...]
> 00:06.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev
> 01)
> 00:07.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev
> 01)

I'm actually surprised that the RHEL-6 guest is not affected by this restriction.

Can you please boot with
- "loglvl=all guest_loglvl=all" on the xen.gz command line,
- "ignore_loglevel" on the dom0 vmlinuz command line,
- and "ignore_loglevel earlyprintk=xenboot" on the RHEL-6 and RHEL-7 guest command lines?

I'd like to compare the PCI initialization in the guests.
Comment 5 Laszlo Ersek 2012-06-29 08:43:55 UTC 
Hi Don,

can you please take a quick look? Could this be related to xen_emul_unplug in RHEL-6?

Thank you,
Laszlo
Comment 6 Wei Shi 2012-06-29 09:45:39 UTC 
(In reply to comment #4)
> Thank you. Interesting. RHEL-5 userspace, "tools/ioemu/vl.h" says:
> 
>   /* PCI slot 6~7 support ACPI PCI hot plug */
>   #define PHP_SLOT_START  (6)
>   #define PHP_SLOT_END    (8)
>   #define PHP_SLOT_LEN    (PHP_SLOT_END - PHP_SLOT_START)
>   #define PHP_TO_PCI_SLOT(x) (x + PHP_SLOT_START)
>   #define PCI_TO_PHP_SLOT(x) (x - PHP_SLOT_START)
>   #define PHP_DEVFN_START (PHP_SLOT_START << 3)
>   #define PHP_DEVFN_END   (PHP_SLOT_END << 3)
> 
> and both of the following (RHEL-7 guest) quotes from comment 0 seem to be
> consistent with that:
> 
> > # xm pci-list [$domU]
> > VSlt domain   bus   slot   func
> > 6      0    3     10     0      
> > 7      0    3     10     1      
> > 0      0    3     10     2      
> > 0      0    3     10     3      
> > [... VSlt=0 in the rest (VSlt is "Virtual Slot") ...]
> 
> and
> 
> > 9.[guest]Check nics in guest by command lspci, only two VF are visible:
> > # lspci
> > [...]
> > 00:06.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev
> > 01)
> > 00:07.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev
> > 01)
> 
> I'm actually surprised that the RHEL-6 guest is not affected by this
> restriction.
> 
> Can you please boot with
> - "loglvl=all guest_loglvl=all" on the xen.gz command line,
> - "ignore_loglevel" on the dom0 vmlinuz command line,
> - and "ignore_loglevel earlyprintk=xenboot" on the RHEL-6 and RHEL-7 guest
> command lines?
> 
> I'd like to compare the PCI initialization in the guests.

After i added "ignore_loglevel earlyprintk=xenboot" to kernel cmdline, both rhel6 and rhel7 guest can't be booted up.
Comment 7 Wei Shi 2012-06-29 09:49:53 UTC 
(In reply to comment #5)
> Hi Don,
> 
> can you please take a quick look? Could this be related to xen_emul_unplug
> in RHEL-6?
> 
> Thank you,
> Laszlo

Yes, it's relate to this parameter, after i adjust both rhel6 and rhel7 to the same value, the lspci resualt is just the same.

PS:
  before i change the value, the xen_emul_unplug parameter's values is:
  rhel6.3--- ide-disks
  rhel7.0--- ide-all
  they set different values.
Comment 8 Laszlo Ersek 2012-06-29 10:05:01 UTC 
So, if you set (non-default) "ide-disks" for rhel7.0, so that it matches rhel6.3, then rhel7.0 works as well?

Similarly, if you set (non-default) "ide-all" for rhel6.3, so that it matches rhel7.0, then rhel6.3 stops working as well? (Meaning, it only sees two VFs?)

Thanks.
Comment 9 Wei Shi 2012-06-29 10:11:09 UTC 
(In reply to comment #8)
> So, if you set (non-default) "ide-disks" for rhel7.0, so that it matches
> rhel6.3, then rhel7.0 works as well?
> 
> Similarly, if you set (non-default) "ide-all" for rhel6.3, so that it
> matches rhel7.0, then rhel6.3 stops working as well? (Meaning, it only sees
> two VFs?)
> 
> Thanks.

Yes, exactly as you say.
Comment 10 Don Dutile (Red Hat) 2012-06-29 14:57:15 UTC 
(In reply to comment #5)
> Hi Don,
> 
> can you please take a quick look? Could this be related to xen_emul_unplug
> in RHEL-6?
> 
> Thank you,
> Laszlo

Well, I see you got your answer(s) in c#7 & c#9.
does the xen_emul_unplug mess up ACPI with too many slots not being hot-pluggable??? ... I'm curious...
Comment 11 Don Dutile (Red Hat) 2012-06-29 15:01:57 UTC 
(In reply to comment #5)
> Hi Don,
> 
> can you please take a quick look? Could this be related to xen_emul_unplug
> in RHEL-6?
> 
> Thank you,
> Laszlo

I just recalled -- there was a bug in the (upstream) xen emul code that would disable VFs inadvertently when 'all' was set.  did that patch not get into rhel5's xen code?  but, 2 VFs make it .... hmmm.... I wonder if the bug exists for all slots after last xen-blk device (0.11.0).
Comment 14 Laszlo Ersek 2012-07-04 11:59:24 UTC 
* I checked the guest configs, they seem correct. The only differences are:
name, disk backing file, and vif MAC.

* (Hints: [1] [2]) In the RHEL-7 guest, I added the following line to
/etc/default/grub:

  GRUB_TERMINAL="serial console"

then ran

  grub2-mkconfig -o /boot/grub2/grub.cfg

which allowed me to edit the kernel command line in xm console / grub also
in the RHEL-7 guest.

* RHEL-6 guest with no xen_emul_unplug:

  [root@dhcp-8-202 ~]# cat /proc/cmdline
    ro root=/dev/mapper/vg_dhcp8202-lv_root rd_NO_LUKS LANG=en_US.UTF-8
    rd_NO_MD rd_LVM_LV=vg_dhcp8202/lv_root SYSFONT=latarcyrheb-sun16
    KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rd_LVM_LV=vg_dhcp8202/lv_swap
    console=tty0 console=ttyS0,115200n8

  [root@dhcp-8-202 ~]# uname -r
    2.6.32-279.el6.x86_64

  [root@dhcp-8-202 ~]# lspci
    [...]
    00:06.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:07.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    [...]

* RHEL-6 guest with xen_emul_unplug=ide-disks:

  [root@dhcp-8-202 ~]# cat /proc/cmdline
    ro root=/dev/mapper/vg_dhcp8202-lv_root rd_NO_LUKS LANG=en_US.UTF-8
    rd_NO_MD rd_LVM_LV=vg_dhcp8202/lv_root SYSFONT=latarcyrheb-sun16
    KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rd_LVM_LV=vg_dhcp8202/lv_swap
    xen_emul_unplug=ide-disks console=tty0 console=ttyS0,115200n8

  [root@dhcp-8-202 ~]# uname -r
    2.6.32-279.el6.x86_64

  [root@dhcp-8-202 ~]# lspci
    [...]
    00:03.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:04.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:05.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:06.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:07.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:08.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:09.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:0a.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:0b.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:0c.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:0d.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:0e.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:0f.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    [...]

* RHEL-7 guest with no xen_emul_unplug:

  [root@rhel7 ~]# cat /proc/cmdline
    BOOT_IMAGE=/vmlinuz-3.3.0-0.15.el7.x86_64
    root=/dev/mapper/vg_rhel7-lv_root ro rd.md=0 rd.dm=0 SYSFONT=True
    rd.lvm.lv=vg_rhel7/lv_root KEYTABLE=us rd.luks=0
    rd.lvm.lv=vg_rhel7/lv_swap LANG=en_US.UTF-8 console=tty0
    console=ttyS0,112500

  [root@rhel7 ~]# uname -r
    3.3.0-0.15.el7.x86_64

  [root@rhel7 ~]# lspci
    [...]
    00:06.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:07.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    [...]

* RHEL-7 guest with xen_emul_unplug=ide-disks:

  [root@rhel7 ~]# cat /proc/cmdline
    BOOT_IMAGE=/vmlinuz-3.3.0-0.15.el7.x86_64
    root=/dev/mapper/vg_rhel7-lv_root ro rd.md=0 rd.dm=0 SYSFONT=True
    rd.lvm.lv=vg_rhel7/lv_root KEYTABLE=us rd.luks=0
    rd.lvm.lv=vg_rhel7/lv_swap LANG=en_US.UTF-8 xen_emul_unplug=ide-disks
    console=tty0 console=ttyS0,112500

  [root@rhel7 ~]# uname -r
    3.3.0-0.15.el7.x86_64

  [root@rhel7 ~]# lspci
    [...]
    00:03.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:04.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:05.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:06.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:07.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:08.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:09.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:0a.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:0b.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:0c.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:0d.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:0e.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:0f.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    00:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function
    (rev 01)
    [...]

* "ide-all" is an invalid value for xen_emul_unplug, both in RHEL-6 and
   RHEL-7.

* Since the RHEL-6 and the RHEL-7 kernels behave identically with the same
xen_emul_unplug setting, and their behaviors match in the default case as
well (ie. with "xen_emul_unplug" absent), I'm closing this as NOTABUG. The
apparently different behavior is caused by whatever composes the kernel
command lines differently.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=801650#c61
[2] http://post-office.corp.redhat.com/archives/tech-list/2011-December/msg00080.html