Bug 1294688 - Kernel Panic with 'intel_iommu=on' Parameter in HP ProLiant SL390s G7 Server
Summary: Kernel Panic with 'intel_iommu=on' Parameter in HP ProLiant SL390s G7 Server
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: kernel
Version: 7.2
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: rc
: 7.3
Assignee: Prarit Bhargava
QA Contact: Jianlin Shi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-12-29 17:09 UTC by Yaniv Liberman
Modified: 2016-01-04 16:47 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-01-04 16:46:29 UTC
Target Upstream Version:

Attachments (Terms of Use)
Includes Kernel panic screenshots and the output of 'lspci -vvv'. (89.85 KB, application/x-gzip)
2015-12-29 17:09 UTC, Yaniv Liberman 		no flags Details
View All Add an attachment (proposed patch, testcase, etc.)

Description Yaniv Liberman 2015-12-29 17:09:46 UTC 
Created attachment 1110295 [details]
Includes Kernel panic screenshots and the output of 'lspci -vvv'.

Description of problem:
Kernel panic with 'intel_iommu=on' parameter in RHEL 7.2 - HP ProLiant SL390s G7 server.

Version-Release number of selected component (if applicable):
3.10.0-327.el7.x86_64

How reproducible:
This can be reproduced by appending the parameter 'intel_iommu=on' to the Kernel in RHEL 7.2, in a HP ProLiant SL390s G7 server.

Steps to Reproduce:
1. The OS needs to be RHEL 7.2
2. vi /etc/default/grub
3. Append 'intel_iommu=on' to 'CMDLINE', and then :wq
4. grub2-mkconfig -o /boot/grub2/grub.cfg
5. Reboot the server
6. Kernel panic

Actual results:
Kernel panic.

Expected results:
The OS to boot normally (this doesn't happen in RHEL 7.1 and below).

Please see the attached file.
Comment 2 Dan Kenigsberg 2015-12-30 09:53:41 UTC 
Yaniv, iirc, the same host booted fine with a rhel-7.1.z kernel. If so, can you provide its version?
Comment 3 Yaniv Liberman 2015-12-30 12:20:45 UTC 
Hi,

Yes, and here you go:

RHEL 7.1, 3.10.0-229.el7.x86_64
Comment 4 Yaniv Liberman 2015-12-30 13:05:58 UTC 
More information:

Before appending "intel_iommu=on" in RHEL 7.1 (3.10.0-229.el7.x86_64),
--
[root@buri08 ~]# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-3.10.0-229.el7.x86_64 root=/dev/mapper/vg0-lv_root ro rd.lvm.lv=vg0/lv_root crashkernel=auto rd.lvm.lv=vg0/lv_swap rhgb quiet vconsole.font=latarcyrheb-sun16 vconsole.keymap=us console=tty0 console=ttyS1,115200
[root@buri08 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp4s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 78:e7:d1:24:7c:d8 brd ff:ff:ff:ff:ff:ff
3: enp4s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 78:e7:d1:24:7c:d9 brd ff:ff:ff:ff:ff:ff
4: enp5s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 78:e7:d1:24:7c:dc brd ff:ff:ff:ff:ff:ff
5: enp5s0d1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 78:e7:d1:24:7c:dd brd ff:ff:ff:ff:ff:ff
    inet 10.35.19.194/22 brd 10.35.19.255 scope global dynamic enp5s0d1
       valid_lft 43055sec preferred_lft 43055sec
    inet6 fe80::7ae7:d1ff:fe24:7cdd/64 scope link 
       valid_lft forever preferred_lft forever
--

And after appending "intel_iommu=on",
--
[root@buri08 ~]# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-3.10.0-229.el7.x86_64 root=/dev/mapper/vg0-lv_root ro rd.lvm.lv=vg0/lv_root crashkernel=auto rd.lvm.lv=vg0/lv_swap rhgb quiet vconsole.font=latarcyrheb-sun16 vconsole.keymap=us console=tty0 console=ttyS1,115200 intel_iommu=on
[root@buri08 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp4s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 78:e7:d1:24:7c:d8 brd ff:ff:ff:ff:ff:ff
3: enp4s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 78:e7:d1:24:7c:d9 brd ff:ff:ff:ff:ff:ff
--

So, as you can see, the 'enp5s0' and 'enp5s0d1' network interfaces, which are the Mellanox NIC [1], are not recognized by the OS anymore after appending the "intel_iommu=on" parameter in RHEL 7.1.

This can be solved by simply removing the "intel_iommu=on" parameter and rebooting the server.

[1],
--
[root@buri08 ~]# lspci -v |grep "Mellanox"
05:00.0 InfiniBand: Mellanox Technologies MT26438 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE Virtualization+] (rev b0)
--
Comment 5 Prarit Bhargava 2016-01-04 16:22:52 UTC 
Hmm ... I installed 7.1, and updated the kernel.  Everything seems to work AFAICT?

[root@hp-sl390s-01 ~]# hostname
hp-sl390s-01.rhts.eng.bos.redhat.com
[root@hp-sl390s-01 ~]# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-3.10.0-327.el7.x86_64 root=/dev/mapper/rhel_hp--sl390s--01-root ro crashkernel=auto console=ttyS1,115200 rd.lvm.lv=rhel_hp-sl390s-01/root rd.lvm.lv=rhel_hp-sl390s-01/swap LANG=en_US.UTF-8 intel_iommu=on
[root@hp-sl390s-01 ~]# dmesg | grep -i iommu
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.10.0-327.el7.x86_64 root=/dev/mapper/rhel_hp--sl390s--01-root ro crashkernel=auto console=ttyS1,115200 rd.lvm.lv=rhel_hp-sl390s-01/root rd.lvm.lv=rhel_hp-sl390s-01/swap LANG=en_US.UTF-8 intel_iommu=on
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-327.el7.x86_64 root=/dev/mapper/rhel_hp--sl390s--01-root ro crashkernel=auto console=ttyS1,115200 rd.lvm.lv=rhel_hp-sl390s-01/root rd.lvm.lv=rhel_hp-sl390s-01/swap LANG=en_US.UTF-8 intel_iommu=on
[    0.000000] Intel-IOMMU: enabled
[    0.982394] dmar: IOMMU 0: reg_base_addr efffe000 ver 1:0 cap c90780106f0462 ecap f0207e
[    8.438663] IOMMU 0 0xefffe000: using Queued invalidation
[    8.441243] IOMMU: Setting RMRR:
[    8.442773] IOMMU: Setting identity map for device 0000:02:00.0 [0xe763e000 - 0xe763ffff]
[    8.446585] IOMMU: Setting identity map for device 0000:02:00.2 [0xe763e000 - 0xe763ffff]
[    8.450462] IOMMU: Setting identity map for device 0000:00:1d.0 [0xe77f5000 - 0xe77fafff]
[    8.454248] IOMMU: Setting identity map for device 0000:00:1d.1 [0xe77f5000 - 0xe77fafff]
[    8.458045] IOMMU: Setting identity map for device 0000:00:1d.2 [0xe77f5000 - 0xe77fafff]
[    8.561789] IOMMU: Setting identity map for device 0000:00:1d.3 [0xe77f5000 - 0xe77fafff]
[    8.565576] IOMMU: Setting identity map for device 0000:02:00.0 [0xe77f5000 - 0xe77fafff]
[    8.569512] IOMMU: Setting identity map for device 0000:02:00.2 [0xe77f5000 - 0xe77fafff]
[    8.573493] IOMMU: Setting identity map for device 0000:02:00.4 [0xe77f5000 - 0xe77fafff]
[    8.577575] IOMMU: Setting identity map for device 0000:00:1d.7 [0xe77fc000 - 0xe77fdfff]
[    8.581405] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    8.583877] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[root@hp-sl390s-01 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp4s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 78:e7:d1:21:6e:78 brd ff:ff:ff:ff:ff:ff
    inet 10.16.67.67/21 brd 10.16.71.255 scope global dynamic enp4s0f0
       valid_lft 86267sec preferred_lft 86267sec
    inet6 2620:52:0:1040:7ae7:d1ff:fe21:6e78/64 scope global dynamic 
       valid_lft 2591937sec preferred_lft 604737sec
    inet6 fe80::7ae7:d1ff:fe21:6e78/64 scope link 
       valid_lft forever preferred_lft forever
3: enp4s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 78:e7:d1:21:6e:79 brd ff:ff:ff:ff:ff:ff
4: enp5s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 78:e7:d1:21:6e:7d brd ff:ff:ff:ff:ff:ff
5: enp5s0d1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 78:e7:d1:21:6e:7e brd ff:ff:ff:ff:ff:ff

The BUG() FWIW, appears to be an issue where the device did not reset.

P.
Comment 6 Prarit Bhargava 2016-01-04 16:45:21 UTC 
With RHEL7.2 ...

[root@hp-sl390s-01 ~]# hostname
hp-sl390s-01.rhts.eng.bos.redhat.com
[root@hp-sl390s-01 ~]# dmesg | grep -i iommu
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.10.0-327.el7.x86_64 root=/dev/mapper/rhel_hp--sl390s--01-root ro crashkernel=auto rd.lvm.lv=rhel_hp-sl390s-01/root rd.lvm.lv=rhel_hp-sl390s-01/swap console=ttyS1,115200 LANG=en_US.UTF-8 intel_iommu=on
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-327.el7.x86_64 root=/dev/mapper/rhel_hp--sl390s--01-root ro crashkernel=auto rd.lvm.lv=rhel_hp-sl390s-01/root rd.lvm.lv=rhel_hp-sl390s-01/swap console=ttyS1,115200 LANG=en_US.UTF-8 intel_iommu=on
[    0.000000] Intel-IOMMU: enabled
[    0.489877] dmar: IOMMU 0: reg_base_addr efffe000 ver 1:0 cap c90780106f0462 ecap f0207e
[    7.487620] IOMMU 0 0xefffe000: using Queued invalidation
[    7.490170] IOMMU: Setting RMRR:
[    7.491825] IOMMU: Setting identity map for device 0000:02:00.0 [0xe763e000 - 0xe763ffff]
[    7.495646] IOMMU: Setting identity map for device 0000:02:00.2 [0xe763e000 - 0xe763ffff]
[    7.499412] IOMMU: Setting identity map for device 0000:00:1d.0 [0xe77f5000 - 0xe77fafff]
[    7.603191] IOMMU: Setting identity map for device 0000:00:1d.1 [0xe77f5000 - 0xe77fafff]
[    7.607014] IOMMU: Setting identity map for device 0000:00:1d.2 [0xe77f5000 - 0xe77fafff]
[    7.610831] IOMMU: Setting identity map for device 0000:00:1d.3 [0xe77f5000 - 0xe77fafff]
[    7.614665] IOMMU: Setting identity map for device 0000:02:00.0 [0xe77f5000 - 0xe77fafff]
[    7.618427] IOMMU: Setting identity map for device 0000:02:00.2 [0xe77f5000 - 0xe77fafff]
[    7.622354] IOMMU: Setting identity map for device 0000:02:00.4 [0xe77f5000 - 0xe77fafff]
[    7.626204] IOMMU: Setting identity map for device 0000:00:1d.7 [0xe77fc000 - 0xe77fdfff]
[    7.629926] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    7.632502] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[root@hp-sl390s-01 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp4s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 78:e7:d1:21:6e:78 brd ff:ff:ff:ff:ff:ff
    inet 10.16.67.67/21 brd 10.16.71.255 scope global dynamic enp4s0f0
       valid_lft 86306sec preferred_lft 86306sec
    inet6 2620:52:0:1040:7ae7:d1ff:fe21:6e78/64 scope global noprefixroute dynamic 
       valid_lft 2591984sec preferred_lft 604784sec
    inet6 fe80::7ae7:d1ff:fe21:6e78/64 scope link 
       valid_lft forever preferred_lft forever
3: enp4s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 78:e7:d1:21:6e:79 brd ff:ff:ff:ff:ff:ff
4: enp5s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 78:e7:d1:21:6e:7d brd ff:ff:ff:ff:ff:ff
5: enp5s0d1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 78:e7:d1:21:6e:7e brd ff:ff:ff:ff:ff:ff

P.
Comment 7 Prarit Bhargava 2016-01-04 16:46:29 UTC 
AFAICT this works?  Closing as WORKSFORME.  

P.
Note You need to log in before you can comment on or make changes to this bug.