Bug 491897

Summary: WARNING: at drivers/net/wireless/iwlwifi/iwl-sta.c:728 iwl_set_dynamic_key+0x1d9/0x443
Product: [Fedora] Fedora Reporter: Matthew Garrett <mjg>
Component: kernelAssignee: John W. Linville <linville>
Status: CLOSED WORKSFORME QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: low    
Version: rawhideCC: dongpo.liu, don-redhat-z6y, jfeeney, kernel-maint, quintela, reinette.chatre
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-04-22 18:12:17 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
ensure key flags set during key set none

Description Matthew Garrett 2009-03-24 15:20:49 UTC
dmesg was full of the following this morning:

------------[ cut here ]------------
WARNING: at drivers/net/wireless/iwlwifi/iwl-sta.c:728 iwl_set_dynamic_key+0x1d9/0x443 [iwlcore]() (Tainted: G        W )
Hardware name: HP Compaq 2510p 
no space for new kewModules linked in: vfat fat ppp_deflate zlib_deflate ppp_async crc_ccitt ppp_generic slhc jfs ext2 ext4 jbd2 crc16 usb_storage tun aes_x86_64 aes_generic fuse rfcomm bridge stp llc bnep sco l2cap sunrpc ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 cpufreq_ondemand acpi_cpufreq freq_table dm_multipath uinput snd_hda_codec_analog snd_hda_intel snd_hda_codec snd_hwdep snd_seq_dummy snd_seq_oss snd_seq_midi_event arc4 ecb snd_seq joydev pata_pcmcia snd_seq_device snd_pcm_oss sdhci_pci iwlagn sdhci snd_mixer_oss iwlcore mmc_core rfkill btusb firewire_ohci lib80211 sierra bluetooth serio_raw yenta_socket snd_pcm ricoh_mmc iTCO_wdt firewire_core crc_itu_t mac80211 usbserial iTCO_vendor_support tpm_infineon pcspkr rsrc_nonstatic tpm snd_timer cfg80211 hp_accel wmi tpm_bios snd soundcore e1000e lis3lv02d snd_page_alloc ata_generic pata_acpi i915 drm i2c_algo_bit i2c_core video output [last unloaded: microcode]
Pid: 10, comm: events/0 Tainted: G        W  2.6.29-0.258.rc8.git2.fc11.x86_64 #1
Call Trace:
 [<ffffffff8104bae3>] warn_slowpath+0xbc/0xf0
 [<ffffffff81071064>] ? print_lock_contention_bug+0x1b/0xe1
 [<ffffffffa01f9c72>] ? iwl_mac_set_key+0x1c3/0x3a2 [iwlagn]
 [<ffffffffa01d0218>] ? iwl_set_dynamic_key+0x14b/0x443 [iwlcore]
 [<ffffffffa01f9c72>] ? iwl_mac_set_key+0x1c3/0x3a2 [iwlagn]
 [<ffffffffa01d02a6>] iwl_set_dynamic_key+0x1d9/0x443 [iwlcore]
 [<ffffffff8106fab8>] ? trace_hardirqs_on_caller+0x1f/0x153
 [<ffffffff8106fbf9>] ? trace_hardirqs_on+0xd/0xf
 [<ffffffffa01f9cf6>] iwl_mac_set_key+0x247/0x3a2 [iwlagn]
 [<ffffffff81156e1f>] ? debugfs_remove_recursive+0x14e/0x159
 [<ffffffffa012ff62>] __ieee80211_key_todo+0x14f/0x240 [mac80211]
 [<ffffffffa0130189>] ? key_todo+0x0/0x10 [mac80211]
 [<ffffffffa013017b>] ieee80211_key_todo+0x1c/0x2a [mac80211]
 [<ffffffffa0130197>] key_todo+0xe/0x10 [mac80211]
 [<ffffffff8105ccb0>] run_workqueue+0xfd/0x1fd
 [<ffffffff8105cc5f>] ? run_workqueue+0xac/0x1fd
 [<ffffffff8106fab8>] ? trace_hardirqs_on_caller+0x1f/0x153
 [<ffffffff8105ce9f>] worker_thread+0xef/0x100
 [<ffffffff81060d46>] ? autoremove_wake_function+0x0/0x39
 [<ffffffff8105cdb0>] ? worker_thread+0x0/0x100
 [<ffffffff8105cdb0>] ? worker_thread+0x0/0x100
 [<ffffffff810609ad>] kthread+0x4d/0x78
 [<ffffffff810126aa>] child_rip+0xa/0x20
 [<ffffffff8100f88b>] ? __switch_to+0x190/0x398
 [<ffffffff81011fbe>] ? restore_args+0x0/0x30
 [<ffffffff81060960>] ? kthread+0x0/0x78
 [<ffffffff810126a0>] ? child_rip+0x0/0x20
---[ end trace 68619c5238d54832 ]---

This is with 2.6.29-0.258.rc8.git2.fc11.x86_64

Comment 1 John W. Linville 2009-03-24 15:38:39 UTC
Looks like it is triggering code from this patch:

   http://lkml.org/lkml/2008/11/25/306

Comment 2 reinette chatre 2009-04-03 21:26:32 UTC
This may be duplicate of http://bugzilla.kernel.org/show_bug.cgi?id=12415

Please ensure that the source you are running has this part of the patch John refers to (it somehow got lost in some kernel versions):

(copy&pasted ... will not apply)

diff --git a/drivers/net/wireless/iwlwifi/iwl-sta.c b/drivers/net/wireless/iwlwifi/iwl-s
index 412f66b..70a8b21 100644
--- a/drivers/net/wireless/iwlwifi/iwl-sta.c
+++ b/drivers/net/wireless/iwlwifi/iwl-sta.c
@@ -480,6 +480,9 @@ void iwl_clear_stations_table(struct iwl_priv *priv)
        priv->num_stations = 0;
        memset(priv->stations, 0, sizeof(priv->stations));
 
+       /* clean ucode key table bit map */
+       priv->ucode_key_table = 0;
+
        spin_unlock_irqrestore(&priv->sta_lock, flags);
 }
 EXPORT_SYMBOL(iwl_clear_stations_table);

Comment 3 John W. Linville 2009-04-06 18:40:04 UTC
The code is in iwl-core.c instead, but the hunk you reference is there.  (Note that F-10 kernels are still based upon 2.6.27.)

Comment 4 reinette chatre 2009-04-10 04:21:22 UTC
Is this easy to reproduce? Any idea what driver could have been doing at the time? Would it be possible to load driver with debugging 0x43fff to get an idea what driver is doing at the time?

Comment 5 reinette chatre 2009-04-10 04:33:36 UTC
Please use debug level 0x443fff instead

Comment 6 reinette chatre 2009-04-21 22:25:28 UTC
Created attachment 340648 [details]
ensure key flags set during key set

Could you please test if this patch fixes the problem?

Comment 7 Matthew Garrett 2009-04-21 23:03:39 UTC
I'm afraid I haven't been able to reproduce the problem since I reported it.

Comment 8 John W. Linville 2009-04-22 18:12:17 UTC
Optimistically closing on the basis of comment 7... :-)

Comment 9 reinette chatre 2009-04-22 18:27:55 UTC
Oh no. This is the _only_ bugreport we have for the warning that (again) landed us on the front page of kerneloops.org. Please do provide feedback if you encounter the issue again.

If users do not submit bug reports for this ... perhaps it is a sign that we can remove that WARN ... or at least just make it an error message ...

Comment 10 Donald Cohen 2009-06-20 15:58:44 UTC
(In reply to comment #9)
> Oh no. This is the _only_ bugreport we have for the warning that (again) landed
> us on the front page of kerneloops.org. Please do provide feedback if you
> encounter the issue again.
I'm probably responsible for at least some of those kerneloops reports.
I figured that meant that it was at least being looked into.
I'm still getting these.
Below is the latest.
Let me know what I can do to help.
I think this started when I upgraded to fc11.
First one seems to be Jun  9 18:38:56
I think fc11 was still rawhide then.  I updated to rawhide, I think June 5
and then managed to make it real fc11 on June 14.  Yum log shows between those only:
Jun 05 21:06:32 Installed: kernel-2.6.29.4-167.fc11.x86_64
Jun 08 09:09:07 Updated: phonon-4.3.1-4.fc11.x86_64
Jun 08 09:09:07 Updated: phonon-backend-xine-4.3.1-4.fc11.x86_64
Jun 12 01:54:38 Updated: at-spi-1.26.0-1.fc11.x86_64
Jun 14 17:48:44 Erased: eclipse-jdt

Does any of that seem relevant?

Looks like the last reboot before June 9 was June 7
Perhaps it took a few days to run out of space for new modules?
Is this a memory leak in kernel space?

========
Jun 20 01:10:43 number11 kernel: ------------[ cut here ]------------
Jun 20 01:10:43 number11 kernel: WARNING: at drivers/net/wireless/iwlwifi/iwl-sta.c:728 iwl_set_dynamic_key+0x1d9/0x443 [iwlcore]() (Tainted: G        W )
Jun 20 01:10:43 number11 kernel: Hardware name: HP Pavilion dv5 Notebook PC
Jun 20 01:10:43 number11 kernel: no space for new kewModules linked in: aes_x86_64 aes_generic fuse ipt_MASQUERADE iptable_nat nf_nat bridge stp llc bnep sco l2cap bluetooth sunrpc ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 cpufreq_ondemand acpi_cpufreq freq_table dm_multipath uinput snd_hda_codec_idt arc4 ecb snd_hda_intel snd_hda_codec iwlagn uvcvideo videodev iwlcore firewire_ohci snd_hwdep snd_pcm firewire_core v4l1_compat iTCO_wdt lib80211 v4l2_compat_ioctl32 iTCO_vendor_support snd_timer i2c_i801 mac80211 snd video pcspkr sdhci_pci hp_accel sdhci lis3lv02d jmb38x_ms mmc_core r8169 joydev memstick soundcore wmi mii snd_page_alloc cfg80211 output crc_itu_t nouveau drm i2c_algo_bit i2c_core [last unloaded: microcode]
Jun 20 01:10:43 number11 kernel: Pid: 10, comm: events/0 Tainted: G        W  2.6.29.4-167.fc11.x86_64 #1
Jun 20 01:10:43 number11 kernel: Call Trace:
Jun 20 01:10:43 number11 kernel: [<ffffffff810489bf>] warn_slowpath+0xbc/0xf0
Jun 20 01:10:43 number11 kernel: [<ffffffff813aa7eb>] ? mutex_lock+0x27/0x38
Jun 20 01:10:43 number11 kernel: [<ffffffff8118672d>] ? inode_doinit_with_dentry+0x419/0x430
Jun 20 01:10:43 number11 kernel: [<ffffffff81186760>] ? selinux_d_instantiate+0x1c/0x1e
Jun 20 01:10:43 number11 kernel: [<ffffffff8117fb97>] ? security_inode_permission+0x21/0x23
Jun 20 01:10:43 number11 kernel: [<ffffffffa01e602a>] iwl_set_dynamic_key+0x1d9/0x443 [iwlcore]
Jun 20 01:10:43 number11 kernel: [<ffffffff81171aad>] ? debugfs_create_file+0x1a6/0x205
Jun 20 01:10:43 number11 kernel: [<ffffffffa021ecbd>] iwl_mac_set_key+0x245/0x3a0 [iwlagn]
Jun 20 01:10:43 number11 kernel: [<ffffffffa015ed68>] __ieee80211_key_todo+0x156/0x24b [mac80211]
Jun 20 01:10:43 number11 kernel: [<ffffffffa015ef8f>] ? key_todo+0x0/0x10 [mac80211]
Jun 20 01:10:43 number11 kernel: [<ffffffffa015ef81>] ieee80211_key_todo+0x1a/0x28 [mac80211]
Jun 20 01:10:43 number11 kernel: [<ffffffffa015ef9d>] key_todo+0xe/0x10 [mac80211]
Jun 20 01:10:43 number11 kernel: [<ffffffff81058e0a>] run_workqueue+0xa7/0x14a
Jun 20 01:10:43 number11 kernel: [<ffffffff81058f99>] worker_thread+0xec/0xfd
Jun 20 01:10:43 number11 kernel: [<ffffffff8105ca4b>] ? autoremove_wake_function+0x0/0x39
Jun 20 01:10:43 number11 kernel: [<ffffffff81058ead>] ? worker_thread+0x0/0xfd
Jun 20 01:10:43 number11 kernel: [<ffffffff81058ead>] ? worker_thread+0x0/0xfd
Jun 20 01:10:43 number11 kernel: [<ffffffff8105c6b5>] kthread+0x4d/0x78
Jun 20 01:10:43 number11 kernel: [<ffffffff8101264a>] child_rip+0xa/0x20
Jun 20 01:10:43 number11 kernel: [<ffffffff81011f67>] ? restore_args+0x0/0x30
Jun 20 01:10:43 number11 kernel: [<ffffffff8105c668>] ? kthread+0x0/0x78
Jun 20 01:10:43 number11 kernel: [<ffffffff81012640>] ? child_rip+0x0/0x20
Jun 20 01:10:43 number11 kernel: ---[ end trace 3bd93aaf29eccffb ]---

Comment 11 reinette chatre 2009-06-22 15:57:01 UTC
(In reply to comment #10)

> Jun 05 21:06:32 Installed: kernel-2.6.29.4-167.fc11.x86_64
> Jun 08 09:09:07 Updated: phonon-4.3.1-4.fc11.x86_64
> Jun 08 09:09:07 Updated: phonon-backend-xine-4.3.1-4.fc11.x86_64
> Jun 12 01:54:38 Updated: at-spi-1.26.0-1.fc11.x86_64
> Jun 14 17:48:44 Erased: eclipse-jdt
> 
> Does any of that seem relevant?
> 

2.6.29.5 contains a patch that addresses this problem. The details are:


commit 485e8e39a85e8178978252cd17f218e6787851ee
Author: Reinette Chatre <reinette.chatre>
Date:   Thu Apr 30 13:56:31 2009 -0700

    iwlwifi: update key flags at time key is set
    
    commit 299f5462087f3bc2141e6bc83ba7e2b15d8a07d2 upstream.
    
    We need to be symmetrical in what is done when key is set and cleared.
    This is important wrt the key flags as they are used during key
    clearing and if they are not set when the key is set the key cannot be
    cleared completely.
    
    This addresses the many occurences of the WARN found in
    iwl_set_tkip_dynamic_key_info() and tracked in
    http://www.kerneloops.org/searchweek.php?search=iwl_set_dynamic_key
    
    If calling iwl_set_tkip_dynamic_key_info()/iwl_remove_dynamic_key()
    pair a few times in a row will cause that we run out of key space.
    This is because the index stored in the key flags is used by
    iwl_remove_dynamic_key() to decide if it should remove the key.
    Unfortunately the key flags, and hence the key index is currently only
    set at the time the key is written to the device (in
    iwl_update_tkip_key()) and _not_ in iwl_set_tkip_dynamic_key_info().
    Fix this by setting flags in iwl_set_tkip_dynamic_key_info().
    
    Signed-off-by: Reinette Chatre <reinette.chatre>
    Signed-off-by: John W. Linville <linville>
    Signed-off-by: Greg Kroah-Hartman <gregkh>

Comment 12 Donald Cohen 2009-06-22 16:22:57 UTC
(In reply to comment #11)
> > Jun 05 21:06:32 Installed: kernel-2.6.29.4-167.fc11.x86_64
> 2.6.29.5 contains a patch that addresses this problem. The details are:
I gather you mean that the next kernel update will solve the problem.
So I should wait for that to appear?
It doesn't seem to be in the normal repo now.
Things seem to be working, as far as I can tell, in spite of all of these messages.  Am I missing something?  Should I just ignore them and expect them to go away next time I reboot to a newer kernel?

Comment 13 reinette chatre 2009-06-22 16:34:40 UTC
(In reply to comment #12)
> (In reply to comment #11)
> > > Jun 05 21:06:32 Installed: kernel-2.6.29.4-167.fc11.x86_64
> > 2.6.29.5 contains a patch that addresses this problem. The details are:
> I gather you mean that the next kernel update will solve the problem.
> So I should wait for that to appear?
> It doesn't seem to be in the normal repo now.
> Things seem to be working, as far as I can tell, in spite of all of these
> messages.  Am I missing something?  Should I just ignore them and expect them
> to go away next time I reboot to a newer kernel?  

From what I understand F-11 already has 2.6.29.5. A different problem we were working on provided http://koji.fedoraproject.org/koji/buildinfo?buildID=106697 as a link to this kernel. Can you try it?

Comment 14 Donald Cohen 2009-06-22 17:04:47 UTC
(In reply to comment #13)
> From what I understand F-11 already has 2.6.29.5. A different problem we were
plain "yum update" doesn't find it for me
Are you suggesting I try something else?
> working on provided http://koji.fedoraproject.org/koji/buildinfo?buildID=106697
> as a link to this kernel. Can you try it?  
Again, I don't see what you want me to do.
I see
 	 kernel-2.6.29.5-191.fc11.x86_64.rpm (info) (download) 
I guess I could download and install that, but what would that do to future yum updates?

Comment 15 reinette chatre 2009-06-22 17:27:30 UTC
(In reply to comment #14)
> (In reply to comment #13)
> > From what I understand F-11 already has 2.6.29.5. A different problem we were
> plain "yum update" doesn't find it for me
> Are you suggesting I try something else?

I am not familiar with fedora internals and releases. I do know that vanilla 2.6.29.5 contains a fix for the warning that you report and I have been told that it (2.6.29.5) is available for fedora. 

> > working on provided http://koji.fedoraproject.org/koji/buildinfo?buildID=106697
> > as a link to this kernel. Can you try it?  
> Again, I don't see what you want me to do.

You reported an issue and I pointed you to a kernel that fixes the issue. I in turn do not understand your question. If this warning is bothering you, which seems to be true because we are discussing it here, then please try that kernel to see if your problem goes away.

> I see
>    kernel-2.6.29.5-191.fc11.x86_64.rpm (info) (download) 
> I guess I could download and install that, but what would that do to future yum
> updates?  

I am not familiar with what your yum updates do to answer this question. What makes you think there will be an issue? The fix can be found in 2.6.29.5 and later kernels.

Comment 16 Donald Cohen 2009-06-22 17:50:24 UTC
(In reply to comment #15)
I understand. 
The choice is to try this kernel now and worry about various possible
negative consequences later or to wait for a later kernel to appear in
the yum repos.
In order to help me understand the costs and benefits, 
- could you answer my earlier question about what may be going wrong now
other than messages in the log?  Are there any consequences outside the
wireless communication?
- would the result of me trying the new kernel help you or other developers significantly?  Would it be almost as good to wait for the kernel to become available in the normal yum repos and then report the result?

Comment 17 reinette chatre 2009-06-22 20:36:20 UTC
(In reply to comment #16)
> (In reply to comment #15)
> I understand. 
> The choice is to try this kernel now and worry about various possible
> negative consequences later or to wait for a later kernel to appear in
> the yum repos.
> In order to help me understand the costs and benefits, 
> - could you answer my earlier question about what may be going wrong now
> other than messages in the log?  Are there any consequences outside the
> wireless communication?
> - would the result of me trying the new kernel help you or other developers
> significantly?  Would it be almost as good to wait for the kernel to become
> available in the normal yum repos and then report the result?  

I think this is your decision to make. From what I understand your wireless is working fine so there does not seem to be an urgent need for you to upgrade your kernel.