Created attachment 781076 [details]
File: dmesg

Microcode error and iwlagn_mac_flush warning on 3.10 kernel.

Description of problem:
Resumed from suspend, got this error as soon as I logged in. 

Version-Release number of selected component:
kernel

Additional info:
reporter:       libreport-2.1.5
cmdline:        BOOT_IMAGE=/vmlinuz-3.10.3-300.fc19.x86_64 root=/dev/mapper/luks-5fcb615b-59b1-4043-857b-4e44e3deffcd ro rd.md=0 rd.lvm=0 rd.dm=0 rd.luks.uuid=luks-5fcb615b-59b1-4043-857b-4e44e3deffcd vconsole.keymap=us rhgb quiet LANG=en_US.UTF-8 i915.i915_enable_rc6=1 i915.i915_enable_fbc=1 i915.lvds_downclock=1
kernel:         3.10.3-300.fc19.x86_64
runlevel:       N 5
type:           Kerneloops

Truncated backtrace:
WARNING: at drivers/net/wireless/iwlwifi/dvm/../iwl-trans.h:691 iwlagn_mac_flush+0x18b/0x1a0 [iwldvm]()
iwl_trans_wait_tx_queue_empty bad state = 0
Modules linked in: ebtable_nat xt_CHECKSUM tun bridge stp llc ppdev parport_pc parport fuse nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE ip6table_nat nf_nat_ipv6 ip6table_mangle ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 iptable_nat nf_nat_ipv4 nf_nat iptable_mangle nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables vsock vfat fat iTCO_wdt iTCO_vendor_support dell_wmi sparse_keymap arc4 mperf coretemp kvm_intel kvm snd_hda_codec_hdmi iwldvm mac80211 dell_laptop dcdbas snd_hda_codec_realtek joydev microcode snd_hda_intel btusb snd_hda_codec snd_hwdep snd_seq bluetooth uvcvideo snd_seq_device videobuf2_vmalloc videobuf2_memops iwlwifi snd_pcm videobuf2_core videodev media cfg80211 lpc_ich i2c_i801 mfd_core rfkill snd_page_alloc snd_timer snd soundcore wmi binfmt_misc uinput dm_crypt crc32_pclmul crc32c_intel i915 ghash_clmulni_intel i2c_algo_bit drm_kms_helper drm i2c_core video
CPU: 0 PID: 3724 Comm: kworker/u16:66 Not tainted 3.10.3-300.fc19.x86_64 #1
Hardware name: Dell Inc.          Dell System XPS L322X/YYYYY, BIOS A06 12/03/2012
Workqueue: phy0 ieee80211_iface_work [mac80211]
 0000000000000009 ffff8801d459dad8 ffffffff81643216 ffff8801d459db10
 ffffffff8105c211 ffff88022a9d1800 ffff88022a9d1848 ffff88022c9ac000
 ffff88022a9d1810 ffff8801a7cab630 ffff8801d459db70 ffffffff8105c27c
Call Trace:
 [<ffffffff81643216>] dump_stack+0x19/0x1b
 [<ffffffff8105c211>] warn_slowpath_common+0x61/0x80
 [<ffffffff8105c27c>] warn_slowpath_fmt+0x4c/0x50
 [<ffffffffa044575b>] iwlagn_mac_flush+0x18b/0x1a0 [iwldvm]
 [<ffffffffa03d832d>] ieee80211_flush_queues+0xbd/0x140 [mac80211]
 [<ffffffffa03c0d21>] __ieee80211_recalc_idle+0xb1/0x100 [mac80211]
 [<ffffffffa03c2344>] ieee80211_recalc_idle+0x14/0x30 [mac80211]
 [<ffffffffa03db937>] ieee80211_free_chanctx+0xc7/0x1d0 [mac80211]
 [<ffffffffa03dc2cb>] __ieee80211_vif_release_channel+0x14b/0x1f0 [mac80211]
 [<ffffffffa03dca2d>] ieee80211_vif_release_channel+0x4d/0x70 [mac80211]
 [<ffffffffa03ece74>] ieee80211_set_disassoc+0x254/0x3b0 [mac80211]
 [<ffffffffa03edeae>] ieee80211_sta_connection_lost.isra.22+0x3e/0x80 [mac80211]
 [<ffffffffa03ef002>] ieee80211_sta_work+0x1d2/0xf70 [mac80211]
 [<ffffffff8117dcce>] ? kfree+0x13e/0x180
 [<ffffffff8152c607>] ? kfree_skbmem+0x37/0x90
 [<ffffffff8117dadc>] ? kmem_cache_free+0x20c/0x220
 [<ffffffffa03c1680>] ? ieee80211_iface_work+0xd0/0x360 [mac80211]
 [<ffffffffa03c1879>] ieee80211_iface_work+0x2c9/0x360 [mac80211]
 [<ffffffff81077c59>] ? pwq_activate_delayed_work+0x39/0x80
 [<ffffffff81079b95>] process_one_work+0x175/0x420
 [<ffffffff8107a7ab>] worker_thread+0x11b/0x3a0
 [<ffffffff8107a690>] ? rescuer_thread+0x340/0x340
 [<ffffffff81080b00>] kthread+0xc0/0xd0
 [<ffffffff81080a40>] ? insert_kthread_work+0x40/0x40
 [<ffffffff8165176c>] ret_from_fork+0x7c/0xb0
 [<ffffffff81080a40>] ? insert_kthread_work+0x40/0x40

Eric, Vinod,

Does the issue happened at random or you can recreate it ?

I don't know how to recreate it, seems to be at random for me.

Here is a different trace I got the next day, maybe it's related?

[153943.658644] ------------[ cut here ]------------
[153943.658678] WARNING: at net/mac80211/key.c:597 ieee80211_free_keys+0x166/0x170 [mac80211]()
[153943.658680] Modules linked in: vhost_net macvtap macvlan tcp_lp rfcomm bnep btusb bluetooth fuse ebtable_nat xt_CHECKSUM tun bridge stp llc rpcsec_gss_krb5 nfsv4 nfs dns_resolver fscache target_core_pscsi target_core_file target_core_iblock iscsi_target_mod target_core_mod nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE ip6table_nat nf_nat_ipv6 ip6table_mangle ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 iptable_nat nf_nat_ipv4 nf_nat iptable_mangle nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel arc4 iwldvm uvcvideo snd_hda_codec iTCO_wdt iTCO_vendor_support snd_hwdep snd_seq snd_seq_device snd_pcm e1000e mac80211 videobuf2_vmalloc videobuf2_memops videobuf2_core videodev media mperf coretemp snd_page_alloc kvm_intel iwlwifi snd_timer thinkpad_acpi snd kvm sdhci_pci sdhci mmc_core cfg80211 ptp soundcore microcode joydev pps_core lpc_ich i2c_i801 mfd_core rfkill wmi tpm_tis tpm tpm_bios nfsd auth_rpcgss nfs_acl lockd sunrpc binfmt_misc uinput dm_crypt usb_storage crc32_pclmul crc32c_intel i915 ghash_clmulni_intel i2c_algo_bit drm_kms_helper drm video i2c_core
[153943.658726] CPU: 1 PID: 808 Comm: NetworkManager Tainted: G        W    3.10.3-300.fc19.x86_64 #1
[153943.658727] Hardware name: LENOVO 2429BQ1/2429BQ1, BIOS G4ET37WW (1.12 ) 05/29/2012
[153943.658728]  0000000000000009 ffff88021152f678 ffffffff81643216 ffff88021152f6b0
[153943.658730]  ffffffff8105c211 ffff88021152f6d0 ffff8802108ac800 ffff8802108acaa8
[153943.658731]  ffff8802108aca90 ffff8802108acaa8 ffff88021152f6c0 ffffffff8105c2ea
[153943.658733] Call Trace:
[153943.658738]  [<ffffffff81643216>] dump_stack+0x19/0x1b
[153943.658741]  [<ffffffff8105c211>] warn_slowpath_common+0x61/0x80
[153943.658743]  [<ffffffff8105c2ea>] warn_slowpath_null+0x1a/0x20
[153943.658751]  [<ffffffffa0482e26>] ieee80211_free_keys+0x166/0x170 [mac80211]
[153943.658758]  [<ffffffffa046e994>] ieee80211_do_stop+0x274/0x7c0 [mac80211]
[153943.658761]  [<ffffffff816490c5>] ? _raw_spin_unlock_bh+0x15/0x20
[153943.658764]  [<ffffffff8155cf79>] ? dev_deactivate_many+0x1f9/0x240
[153943.658771]  [<ffffffffa046eefa>] ieee80211_stop+0x1a/0x20 [mac80211]
[153943.658773]  [<ffffffff815386cd>] __dev_close_many+0x7d/0xc0
[153943.658774]  [<ffffffff81538753>] __dev_close+0x43/0x70
[153943.658777]  [<ffffffff8153f232>] __dev_change_flags+0x92/0x170
[153943.658778]  [<ffffffff8153f3ad>] dev_change_flags+0x1d/0x60
[153943.658780]  [<ffffffff8154c8e9>] do_setlink+0x339/0xa00
[153943.658782]  [<ffffffff815e7e5a>] ? inet6_fill_link_af+0x1a/0x30
[153943.658785]  [<ffffffff813058b2>] ? nla_parse+0x32/0xe0
[153943.658787]  [<ffffffff8154dc74>] rtnl_newlink+0x394/0x5e0
[153943.658789]  [<ffffffff8128a86e>] ? selinux_capable+0x2e/0x40
[153943.658791]  [<ffffffff8154a7c9>] rtnetlink_rcv_msg+0x99/0x260
[153943.658793]  [<ffffffff81287935>] ? sock_has_perm+0x75/0x90
[153943.658795]  [<ffffffff8154a730>] ? rtnetlink_rcv+0x30/0x30
[153943.658797]  [<ffffffff81568da9>] netlink_rcv_skb+0xa9/0xc0
[153943.658798]  [<ffffffff8154a728>] rtnetlink_rcv+0x28/0x30
[153943.658800]  [<ffffffff8156847d>] netlink_unicast+0xdd/0x190
[153943.658801]  [<ffffffff815687f6>] netlink_sendmsg+0x2c6/0x6c0
[153943.658803]  [<ffffffff81565702>] ? netlink_seq_start+0x2/0x40
[153943.658804]  [<ffffffff815668b4>] ? netlink_recvmsg+0x204/0x390
[153943.658807]  [<ffffffff81525d29>] sock_sendmsg+0x99/0xd0
[153943.658809]  [<ffffffff815263c8>] ? sock_recvmsg+0xa8/0xe0
[153943.658810]  [<ffffffff815dfbb7>] ? unix_dgram_sendmsg+0x547/0x600
[153943.658812]  [<ffffffff8152614e>] ___sys_sendmsg+0x39e/0x3b0
[153943.658814]  [<ffffffff815262de>] ? SYSC_sendto+0x17e/0x1c0
[153943.658816]  [<ffffffff81526f32>] __sys_sendmsg+0x42/0x80
[153943.658817]  [<ffffffff81526f82>] SyS_sendmsg+0x12/0x20
[153943.658820]  [<ffffffff81651819>] system_call_fastpath+0x16/0x1b
[153943.658821] ---[ end trace b47ef897b0e85132 ]---

I can't recreate at will either.

However, there is a correlation, possibly strong, with use in a corporate WiFi network with WPA2 Enterprise + authentication=PEAP. In general, the Intel Centrino Advanced-N 6235 in this machine tends to have occasional microcode SW errors and/or 'fail to flush all tx fifo queues' errors. In other environments with "weaker" security, it's relatively more stable. So, the overall poor experience I have with this chipset in this environment seems to be a common theme, and this was true under F18.

Another F19 laptop I use with the Intel Ultimate-N 6300 chipset in the same environment is (knock on wood) far more stable.

I'll try to add more info about the microcode SW errors when I get a chance, as well as similar stack dumps that follow those events.

NB: bug 979873 is related.

Created attachment 783647 [details]
snippet from dmesg showing microcode errors and kernel stackdump

Let's try to grab verbose debug logs. File with logs would be big, but I think we have no other choice to debug this issue. Please do:

Add this line in /etc/rsyslog.conf: 
kern.*                                                  /var/log/kernel

Restart rsyslog:
# systemctl restart rsyslog.service

Reload module with verbose debug option to grab messages:
# modprobe -r iwldvm
# echo > /var/log/kernel
# modprobe iwlwifi debug=0x47ffffff

If you plan to restart system you need do above steps on system start (can be added to some init script).

When the problem happen, please compress /var/log/kernel file and attach it here (or if too big provide somewhere to download).

(In reply to Vinod Kutty from comment #6)
> However, there is a correlation, possibly strong, with use in a corporate
> WiFi network with WPA2 Enterprise + authentication=PEAP. In general, the

If bug depend on encryption method, possibly it can be workaround by swcrypto=1 module option. But before start to use that option please try to grab verbose debug logs.

(In reply to Stanislaw Gruszka from comment #10)
> (In reply to Vinod Kutty from comment #6)
> > However, there is a correlation, possibly strong, with use in a corporate
> > WiFi network with WPA2 Enterprise + authentication=PEAP. In general, the
> 
> If bug depend on encryption method, possibly it can be workaround by
> swcrypto=1 module option. But before start to use that option please try to
> grab verbose debug logs.

Have tried this before in F18 with no luck. Agreed, we can defer this until after we try verbose logging.

Description of problem:
Just doing work and the wireless drops out of nowhere.

Version-Release number of selected component:
kernel

Additional info:
reporter:       libreport-2.1.6
cmdline:        BOOT_IMAGE=/vmlinuz-3.10.5-201.fc19.x86_64 root=/dev/mapper/fedora_cslam0358-root ro rd.md=0 rd.dm=0 vconsole.keymap=us rd.lvm.lv=fedora_cslam0358/swap rd.lvm.lv=fedora_cslam0358/root rd.luks=0 vconsole.font=latarcyrheb-sun16 rhgb quiet LANG=en_US.UTF-8
kernel:         3.10.5-201.fc19.x86_64
runlevel:       N 5
type:           Kerneloops

Truncated backtrace:
WARNING: at drivers/net/wireless/iwlwifi/dvm/../iwl-trans.h:691 iwlagn_mac_flush+0x18b/0x1a0 [iwldvm]()
iwl_trans_wait_tx_queue_empty bad state = 0
Modules linked in: nls_utf8 hfsplus usb_storage fuse ebtable_nat xt_CHECKSUM nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE ip6table_nat nf_nat_ipv6 ip6table_mangle ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 tun iptable_nat nf_nat_ipv4 nf_nat bridge stp llc iptable_mangle nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack rfcomm ebtable_filter ebtables ip6table_filter bnep ip6_tables vfat fat snd_hda_codec_hdmi snd_hda_codec_realtek iTCO_wdt iTCO_vendor_support mperf snd_hda_intel coretemp snd_hda_codec snd_hwdep kvm_intel snd_seq snd_seq_device arc4 kvm crc32_pclmul crc32c_intel ghash_clmulni_intel btusb microcode iwldvm bluetooth joydev uvcvideo mac80211 videobuf2_vmalloc videobuf2_memops videobuf2_core videodev i2c_i801 media sdhci_pci sdhci iwlwifi snd_pcm lpc_ich cfg80211 mfd_core mmc_core e1000e mei_me ptp mei pps_core snd_page_alloc snd_timer thinkpad_acpi snd soundcore tpm_tis rfkill tpm tpm_bios uinput binfmt_misc nouveau mxm_wmi ttm firewire_ohci i2c_algo_bit drm_kms_helper firewire_core crc_itu_t drm i2c_core wmi video
CPU: 2 PID: 8166 Comm: kworker/u16:0 Not tainted 3.10.5-201.fc19.x86_64 #1
Hardware name: LENOVO 2359CTO/2359CTO, BIOS G4ET94WW (2.54 ) 05/23/2013
Workqueue: phy0 ieee80211_csa_connection_drop_work [mac80211]
 0000000000000009 ffff8801d8fa5be8 ffffffff81638f56 ffff8801d8fa5c20
 ffffffff8105c211 ffff8803feaa9800 ffff8803feaa9848 ffff880406a14000
 ffff8803feaa9810 ffff8803b3f729f0 ffff8801d8fa5c80 ffffffff8105c27c
Call Trace:
 [<ffffffff81638f56>] dump_stack+0x19/0x1b
 [<ffffffff8105c211>] warn_slowpath_common+0x61/0x80
 [<ffffffff8105c27c>] warn_slowpath_fmt+0x4c/0x50
 [<ffffffffa04f075b>] iwlagn_mac_flush+0x18b/0x1a0 [iwldvm]
 [<ffffffffa03ed32d>] ieee80211_flush_queues+0xbd/0x140 [mac80211]
 [<ffffffffa03d5d21>] __ieee80211_recalc_idle+0xb1/0x100 [mac80211]
 [<ffffffffa03d7344>] ieee80211_recalc_idle+0x14/0x30 [mac80211]
 [<ffffffffa03f0937>] ieee80211_free_chanctx+0xc7/0x1d0 [mac80211]
 [<ffffffffa03f12cb>] __ieee80211_vif_release_channel+0x14b/0x1f0 [mac80211]
 [<ffffffffa03f1a2d>] ieee80211_vif_release_channel+0x4d/0x70 [mac80211]
 [<ffffffffa0401e94>] ieee80211_set_disassoc+0x254/0x3b0 [mac80211]
 [<ffffffffa0402f66>] __ieee80211_disconnect+0x56/0xc0 [mac80211]
 [<ffffffffa0402fe5>] ieee80211_csa_connection_drop_work+0x15/0x20 [mac80211]
 [<ffffffff81079b95>] process_one_work+0x175/0x420
 [<ffffffff8107a7ab>] worker_thread+0x11b/0x3a0
 [<ffffffff8107a690>] ? rescuer_thread+0x340/0x340
 [<ffffffff81080b00>] kthread+0xc0/0xd0
 [<ffffffff81080a40>] ? insert_kthread_work+0x40/0x40
 [<ffffffff816474ac>] ret_from_fork+0x7c/0xb0
 [<ffffffff81080a40>] ? insert_kthread_work+0x40/0x40

(In reply to Stanislaw Gruszka from comment #9)

> Reload module with verbose debug option to grab messages:
> # modprobe -r iwldvm
> # echo > /var/log/kernel
> # modprobe iwlwifi debug=0x47ffffff

I tried the above last week, but I can no longer connect to my network. I end up encountering this:

----
Aug 14 09:19:34 eval-dell-xps-l322x kernel: [ 3600.110420] wlp1s0: authenticate with 16:06:01:xx:yy:zz
Aug 14 09:19:34 eval-dell-xps-l322x kernel: [ 3600.117624] wlp1s0: send auth to 16:06:01:xx:yy:zz(try 1/3)
Aug 14 09:19:34 eval-dell-xps-l322x kernel: [ 3600.123280] wlp1s0: authenticated
Aug 14 09:19:34 eval-dell-xps-l322x NetworkManager[607]: <info> (wlp1s0): supplicant interface state: scanning -> authenticating
Aug 14 09:19:39 eval-dell-xps-l322x kernel: [ 3605.126761] wlp1s0: deauthenticating from 16:06:01:xx:yy:zz by local choice (reason=3)
Aug 14 09:19:39 eval-dell-xps-l322x NetworkManager[607]: <info> (wlp1s0): supplicant interface state: authenticating -> disconnected
----

which results NetworkManager prompting me again for my password and then looping through the same sequence. Is it some sort of race condition with NetworkManager not realizing the auth was successful? I'm somewhat ignorant of the details ...

Had to turn debug off to get WiFi connectivity.

Hmm, that's strange . Did you try to restart NM i.e. "systemctl restart NetworkManager" ? If restart do not help, please add those lines at the end of rsyslog.conf

$SystemLogRateLimitInterval 0
*.*                /var/log/everything.log

Modify OTHER_ARGS to add -s and -dd option in /etc/sysconfig/wpa_supplicant file. For example OTHER_ARGS should looks like this: "OTHER_ARGS="-s -u -f -dd /var/log/wpa_supplicant.log -P /var/run/wpa_supplicant.pid"

Then turn on iwlwifi debugging to reproduce this authentication issue and provide everything.log file.

*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 19 kernel bugs.

Fedora 19 has now been rebased to 3.11.1-200.fc19.  Please test this kernel update and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you experience different issues, please open a new bug report for those.

This bug is being closed with INSUFFICIENT_DATA as there has not been a response in 2 weeks. If you are still experiencing this issue, please reopen and attach the relevant data from the latest kernel you are running and any data that might have been requested previously.