Bug 1203241

Summary:	DMA: Out of SW-IOMMU space for 84 bytes at device 0000:01:00.0
Product:	[Fedora] Fedora	Reporter:	Joergen Thomsen <jth>
Component:	kernel	Assignee:	Kernel Maintainer List <kernel-maint>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	21	CC:	gansalmon, itamar, jonathan, kernel-maint, madhu.chinakonda, mchehab
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-05-29 15:39:11 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Joergen Thomsen 2015-03-18 12:54:43 UTC

Description of problem:
After an update to kernel 3.18.9-100.fc20 numerous Out of SW-IOMMU  and Failed to map TX DMA messages occur. Networking bounces up and down (probably getting up again only by a local script testing and restarting failing network connections)


Version-Release number of selected component (if applicable):
3.18.9-100.fc20

How reproducible:
constant

Steps to Reproduce:
1. boot the server and wait a few minutes
2.
3.

Actual results:

 DMA: Out of SW-IOMMU space for 84 bytes at device 0000:01:00.0

Expected results:
 No outage

Additional info:

Mar 18 10:28:59 ns4 kernel: wlan0: No active IBSS STAs - trying to scan for other IBSS networks with same SSID (merge)
Mar 18 10:29:30 ns4 kernel: wlan0: No active IBSS STAs - trying to scan for other IBSS networks with same SSID (merge)
Mar 18 10:29:37 ns4 kernel: rtl8192ce 0000:01:00.0: swiotlb buffer is full (sz: 9100 bytes)
Mar 18 10:29:37 ns4 kernel: DMA: Out of SW-IOMMU space for 9100 bytes at device 0000:01:00.0
Mar 18 10:29:44 ns4 kernel: rtl8192ce 0000:01:00.0: swiotlb buffer is full (sz: 9100 bytes)
Mar 18 10:29:44 ns4 kernel: DMA: Out of SW-IOMMU space for 9100 bytes at device 0000:01:00.0


Mar 18 10:36:20 ns4 kernel: r8169 0000:02:00.0: swiotlb buffer is full (sz: 634 bytes)
Mar 18 10:36:20 ns4 kernel: DMA: Out of SW-IOMMU space for 634 bytes at device 0000:02:00.0
Mar 18 10:36:20 ns4 kernel: r8169 0000:02:00.0 eth0: Failed to map TX DMA!
Mar 18 10:36:20 ns4 kernel: r8169 0000:02:00.0: swiotlb buffer is full (sz: 1442 bytes)
Mar 18 10:36:20 ns4 kernel: DMA: Out of SW-IOMMU space for 1442 bytes at device 0000:02:00.0
Mar 18 10:36:20 ns4 kernel: r8169 0000:02:00.0 eth0: Failed to map TX DMA!
Mar 18 10:36:20 ns4 kernel: r8169 0000:02:00.0: swiotlb buffer is full (sz: 646 bytes)
Mar 18 10:36:20 ns4 kernel: DMA: Out of SW-IOMMU space for 646 bytes at device 0000:02:00.0
Mar 18 10:36:20 ns4 kernel: r8169 0000:02:00.0 eth0: Failed to map TX DMA!
Mar 18 10:36:20 ns4 kernel: rtl8192ce 0000:01:00.0: swiotlb buffer is full (sz: 84 bytes)

[    0.104033] pci 0000:01:00.0: [10ec:8176] type 00 class 0x028000
[    0.104077] pci 0000:01:00.0: reg 0x10: [io  0xe000-0xe0ff]
[    0.104146] pci 0000:01:00.0: reg 0x18: [mem 0xf7b00000-0xf7b03fff 64bit]
[    0.104398] pci 0000:01:00.0: supports D1 D2
[    0.104400] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.104455] pci 0000:01:00.0: System wakeup disabled by ACPI

[    2.007904] r8169 0000:02:00.0: irq 28 for MSI/MSI-X
[    2.008420] r8169 0000:02:00.0 eth0: RTL8168g/8111g at 0xffffc90000c6a000, 80:ee:73:83:4e:b2, XID 0c000800 IRQ 28
[    2.008423] r8169 0000:02:00.0 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko]

Fallback to kernel 3.18.7-100.fc20 appears to solve the problem

Comment 1 Joergen Thomsen 2015-03-18 13:21:58 UTC

Fallback to 3.18.7-100.fc20 was not sufficient :(

[ 3866.624983] rtl8192ce 0000:01:00.0: swiotlb buffer is full (sz: 9100 bytes)
[ 3866.624989] DMA: Out of SW-IOMMU space for 9100 bytes at device 0000:01:00.0
[ 3875.311219] wlan0: No active IBSS STAs - trying to scan for other IBSS networks with same SSID (merge)
[ 3906.342003] wlan0: No active IBSS STAs - trying to scan for other IBSS networks with same SSID (merge)
[ 3967.251814] wlan0: No active IBSS STAs - trying to scan for other IBSS networks with same SSID (merge)
[ 4028.289697] wlan0: No active IBSS STAs - trying to scan for other IBSS networks with same SSID (merge)
[ 4039.880797] rtl8192ce 0000:01:00.0: swiotlb buffer is full (sz: 9100 bytes)
[ 4039.880802] DMA: Out of SW-IOMMU space for 9100 bytes at device 0000:01:00.0
[ 4089.199564] wlan0: No active IBSS STAs - trying to scan for other IBSS networks with same SSID (merge)

Comment 2 Joergen Thomsen 2015-03-18 16:53:45 UTC

It started after 3.17.7-200.fc20.x86_64

Comment 3 Joergen Thomsen 2015-03-31 22:10:06 UTC

After upgrading to Fedora 21 the problem is confirmed in 
3.19.1-201.fc21.x86_64 #1 SMP Wed Mar 18 04:29:24 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux as well

Now with a total kernel crash / freeze requiring a cold reboot !


Mar 31 23:34:01 ns4 kernel: DMA: Out of SW-IOMMU space for 9100 bytes at device 0000:01:00.0
Mar 31 23:34:01 ns4 kernel: rtl8192ce 0000:01:00.0: swiotlb buffer is full (sz: 9100 bytes)
Mar 31 23:34:01 ns4 kernel: DMA: Out of SW-IOMMU space for 9100 bytes at device 0000:01:00.0
Mar 31 23:34:01 ns4 kernel: rtl8192ce 0000:01:00.0: swiotlb buffer is full (sz: 9100 bytes)
Mar 31 23:34:01 ns4 kernel: DMA: Out of SW-IOMMU space for 9100 bytes at device 0000:01:00.0
Mar

Comment 4 Joergen Thomsen 2015-03-31 22:55:29 UTC

Attempting fallback to kernel 3.17.7-200.fc20.x86_64
This is a total showstopper.

Apr  1 00:35:47 ns4 kernel: rtl8192ce 0000:01:00.0: swiotlb buffer is full (sz: 9100 bytes)
Apr  1 00:35:47 ns4 kernel: DMA: Out of SW-IOMMU space for 9100 bytes at device 0000:01:00.0
Apr  1 00:35:47 ns4 kernel: rtl8192ce 0000:01:00.0: swiotlb buffer is full (sz: 9100 bytes)
Apr  1 00:35:47 ns4 kernel: DMA: Out of SW-IOMMU space for 9100 bytes at device 0000:01:00.0
Apr  1 00:35:47 ns4 kernel: rtl8192ce 0000:01:00.0: swiotlb buffer is full (sz: 9100 bytes)
Apr  1 00:35:47 ns4 kernel: DMA: Out of SW-IOMMU space for 9100 bytes at device 0000:01:00.0
Apr  1 00:35:47 ns4 kernel: rtl8192ce 0000:01:00.0: swiotlb buffer is full (sz: 9100 bytes)
Apr  1 00:35:47 ns4 kernel: DMA: Out of SW-IOMMU space for 9100 bytes at device 0000:01:00.0
Apr  1 00:35:47 ns4 kernel: rtl8192ce 0000:01:00.0: swiotlb buffer is full (sz: 9100 bytes)
Apr  1 00:35:47 ns4 kernel: DMA: Out of SW-IOMMU space for 9100 bytes at device 0000:01:00.0
Apr  1 00:35:54 ns4 kernel: BUG: unable to handle kernel NULL pointer dereference at 000000000000011d
Apr  1 00:35:54 ns4 kernel: IP: [<ffffffff81260237>] ep_send_events_proc+0xa7/0x1c0
Apr  1 00:35:54 ns4 kernel: PGD d1eaa067 PUD d1eab067 PMD 0
Apr  1 00:35:54 ns4 kernel: Oops: 0000 [#1] SMP
Apr  1 00:35:54 ns4 kernel: Modules linked in: xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 sit authenc cmac rmd160 crypto_null ip_vti ip_tunnel af_key ah6 ah4 esp6 esp4 xfrm4_mode_beet xfrm4_tunnel tunnel4 xfrm4_mode_tunnel xfrm4_mode_transport xfrm6_mode_transport xfrm6_mode_ro xfrm6_mode_beet xfrm6_mode_tunnel ipcomp ipcomp6 xfrm6_tunnel tunnel6 xfrm_ipcomp camellia_generic camellia_x86_64 cast6_generic cast5_generic cast_common deflate cts gcm ccm serpent_sse2_x86_64 serpent_generic blowfish_generic blowfish_x86_64 blowfish_common twofish_generic twofish_x86_64_3way twofish_x86_64 twofish_common xcbc sha256_ssse3 sha512_ssse3 sha512_generic des3_ede_x86_64 des_generic tpm_rng timeriomem_rng virtio_rng virtio_ring virtio 8021q garp mrp nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_ftp ip6t_rpfilter
Apr  1 00:35:54 ns4 kernel: ip6t_REJECT nf_reject_ipv6 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw intel_rapl iosf_mbi arc4 rtl8192ce x86_pkg_temp_thermal rtl_pci rtl8192c_common coretemp rtlwifi mac80211 kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel cfg80211 rfkill ghash_clmulni_intel iTCO_wdt iTCO_vendor_support i2c_i801 tpm_tis snd_hda_codec_hdmi tpm snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller lpc_ich mfd_core snd_hda_codec snd_hwdep mei_me snd_seq mei snd_seq_device snd_pcm snd_timer
Apr  1 00:35:54 ns4 kernel: snd shpchp soundcore uas usb_storage i915 i2c_algo_bit drm_kms_helper drm r8169 mii video
Apr  1 00:35:54 ns4 kernel: CPU: 0 PID: 1646 Comm: named Not tainted 3.19.1-201.fc21.x86_64 #1
Apr  1 00:35:54 ns4 kernel: Hardware name: Shuttle Inc. DS437/FS437, BIOS 1.01 09/25/2013
Apr  1 00:35:54 ns4 kernel: task: ffff880036b7f440 ti: ffff8800d1c18000 task.ti: ffff8800d1c18000
Apr  1 00:35:54 ns4 kernel: RIP: 0010:[<ffffffff81260237>]  [<ffffffff81260237>] ep_send_events_proc+0xa7/0x1c0
Apr  1 00:35:54 ns4 kernel: RSP: 0018:ffff8800d1c1bd78  EFLAGS: 00010246
Apr  1 00:35:54 ns4 kernel: RAX: 00000000000000d5 RBX: 0000000000000000 RCX: dead000000200200
Apr  1 00:35:54 ns4 kernel: RDX: ffff8800d1c1be10 RSI: ffff8800d1c1bda0 RDI: ffff8801efef4000
Apr  1 00:35:54 ns4 kernel: RBP: ffff8800d1c1bde8 R08: ffff880212ce5298 R09: 0000000000000000
Apr  1 00:35:54 ns4 kernel: R10: 0000000000000003 R11: 0000000000000020 R12: ffff8800d1c1be10
Apr  1 00:35:54 ns4 kernel: R13: 00007f83164d2010 R14: ffff8800d1c1beb8 R15: ffff880212ce5298
Apr  1 00:35:54 ns4 kernel: FS:  00007f8311e60700(0000) GS:ffff88021f200000(0000) knlGS:0000000000000000
Apr  1 00:35:54 ns4 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr  1 00:35:54 ns4 kernel: CR2: 000000000000011d CR3: 00000000d1ea9000 CR4: 00000000001407f0
Apr  1 00:35:54 ns4 kernel: Stack:
Apr  1 00:35:54 ns4 kernel: ffff880036b7f9c8 ffff8800d1ea8360 ffff8800d1ea8300 ffff88021f214580
Apr  1 00:35:54 ns4 kernel: ffff880213994400 0000000000000000 0000000000000019 00000000a5ce9c93
Apr  1 00:35:54 ns4 kernel: ffff8800d1c1be28 ffff8800d1ea8360 ffff8800d1c1be10 ffff8800d1ea8300
Apr  1 00:35:54 ns4 kernel: Call Trace:
Apr  1 00:35:54 ns4 kernel: [<ffffffff81260190>] ? ep_read_events_proc+0xd0/0xd0
Apr  1 00:35:54 ns4 kernel: [<ffffffff812609fa>] ep_scan_ready_list+0x9a/0x200
Apr  1 00:35:54 ns4 kernel: [<ffffffff81260d61>] ep_poll+0x1d1/0x450
Apr  1 00:35:54 ns4 kernel: [<ffffffff810c82e0>] ? wake_up_state+0x20/0x20
Apr  1 00:35:54 ns4 kernel: [<ffffffff81262435>] SyS_epoll_wait+0xb5/0xe0
Apr  1 00:35:54 ns4 kernel: [<ffffffff8113dc16>] ? __audit_syscall_exit+0x1f6/0x2a0
Apr  1 00:35:54 ns4 kernel: [<ffffffff81774029>] system_call_fastpath+0x12/0x17
Apr  1 00:35:54 ns4 kernel: Code: 48 89 c7 e8 0c 8c 27 00 4c 89 ff e8 14 04 15 00 41 8b 47 58 49 8b 7f 18 48 8d 75 b8 4d 89 3f 4d 89 7f 08 48 89 45 c0 48 8b 47 28 <ff> 50 48 41 23 47 58 74 a0 31 d2 66 66 90 41 89 45 00 66 66 90
Apr  1 00:35:54 ns4 kernel: RIP  [<ffffffff81260237>] ep_send_events_proc+0xa7/0x1c0
Apr  1 00:35:54 ns4 kernel: RSP <ffff8800d1c1bd78>
Apr  1 00:35:54 ns4 kernel: CR2: 000000000000011d
Apr  1 00:35:54 ns4 kernel: ---[ end trace 0f7c0965751f1828 ]---
Apr  1 00:35:55 ns4 abrt-dump-journal-oops: abrt-dump-journal-oops: Found oopses: 1
Apr  1 00:35:55 ns4 abrt-dump-journal-oops: abrt-dump-journal-oops: Creating problem directories
Apr  1 00:35:55 ns4 abrt-server: Looking for kernel package
Apr  1 00:35:56 ns4 abrt-dump-journal-oops: Reported 1 kernel oopses to Abrt
Apr  1 00:35:56 ns4 abrt-server: Kernel package kernel-core-3.19.1-201.fc21.x86_64 found

Comment 5 Fedora Kernel Team 2015-04-28 18:28:44 UTC

*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 21 kernel bugs.

Fedora 21 has now been rebased to 3.19.5-200.fc21.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you have moved on to Fedora 22, and are still experiencing this issue, please change the version to Fedora 22.

If you experience different issues, please open a new bug report for those.

Comment 6 Joergen Thomsen 2015-04-29 13:34:00 UTC

Hopefully solved by this

https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?id=be0b5e635883678bfbc695889772fed545f3427d

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=be0b5e635883678bfbc695889772fed545f3427d

Waiting for inclusion in new kernel.

Comment 7 Joergen Thomsen 2015-05-29 15:39:11 UTC

Solved as of kernel 3.19.5