Bug 1582407

hello,

reading the log, I think that the wireless NIC on your computer is not ready for PMF:

May 24 23:07:11 localhost wpa_supplicant[2828]: wlp2s0: WPA: Failed to configure IGTK to the driver
May 24 23:07:11 localhost kernel: wlp2s0: deauthenticating from <router MAC> by local choice (Reason: 1=UNSPECIFIED)
May 24 23:07:11 localhost wpa_supplicant[2828]: wlp2s0: RSN: Failed to configure IGTK

It would be great if you post the output of

# lshw -c net

so we understand if something can be done on the wireless driver. In the meanwhile, can you try disabling PMF on the supplicant (using 2.6-15 version)?

# nmcli connection edit <your connection>
> set 802-11-wireless-security.pmf disable
> save
> activate

thank you in advance!
-- 
davide

Relevant output of $ lshw -c net

  *-network
       description: Wireless interface
       product: Wireless 8260
       vendor: Intel Corporation
       physical id: 0
       bus info: pci@0000:02:00.0
       logical name: wlp2s0
       version: 3a
       serial: <numbers>
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress bus_master cap_list ethernet physical wireless
       configuration: broadcast=yes driver=iwlwifi driverversion=4.16.11-300.fc28.x86_64 firmware=34.0.1 ip=192.168.0.32 latency=0 link=yes multicast=yes wireless=IEEE 802.11
       resources: irq:41 memory:f0c00000-f0c01fff


Editing the connection as you instructed ~did~ work, but since my NIC does support PMF, I don't consider this an ideal solution.

(In reply to kwkaess from comment #2)
 
> Editing the connection as you instructed ~did~ work, but since my NIC does
> support PMF, I don't consider this an ideal solution.

sure, that's not a solution. But with a working wireless connection you are eligible for testing another kernel / another linux-firmware (and in the meanwhile we can try debugging more in deep).

Is that ok for you?

Sounds good.

(In reply to Davide Caratti from comment #3)
> (In reply to kwkaess from comment #2)
>  
> > Editing the connection as you instructed ~did~ work, but since my NIC does
> > support PMF, I don't consider this an ideal solution.
> 
> sure, that's not a solution. But with a working wireless connection you are
> eligible for testing another kernel / another linux-firmware (and in the
> meanwhile we can try debugging more in deep).
> 
> Is that ok for you?

Hi Davide,

We have about 5 people experiencing this issue now as well. Is there any additional debugging that is required? Or are we pretty happy we know what the issue is here?

Happy to help, in the mean-time we'll keep an eye out for a package coming into updates-testing addressing this issue.

(In reply to Brendan Shephard from comment #5)
> (In reply to Davide Caratti from comment #3)
> > (In reply to kwkaess from comment #2)
> >  
> > > Editing the connection as you instructed ~did~ work, but since my NIC does
> > > support PMF, I don't consider this an ideal solution.
> > 
> > sure, that's not a solution. But with a working wireless connection you are
> > eligible for testing another kernel / another linux-firmware (and in the
> > meanwhile we can try debugging more in deep).
> > 
> > Is that ok for you?
> 
> Hi Davide,
> 
> We have about 5 people experiencing this issue now as well. Is there any
> additional debugging that is required? Or are we pretty happy we know what
> the issue is here?

all of you 5 are using Intel 8260, and all of you are seeing failures when the IGTK is installed (like in comment #1) ?
 
> Happy to help, in the mean-time we'll keep an eye out for a package coming
> into updates-testing addressing this issue.

if all of you are seeing the same error message as the one in comment #1, then most probably the fix is in the kernel, not in the wpa_supplicant package [*].

While inspecting commits in 'linux-wireless' treee to find if some commit can be missing from fedora kernel, I spotted this one:

commit 4883145a8e17fd0c232a80dbc212eaebf57b061d
Author: Emmanuel Grumbach <emmanuel.grumbach>
Date:   Mon Jan 29 10:00:05 2018 +0200

    iwlwifi: mvm: set the MFP flag for keys that are used by MFP stations
    
unfortunately I don't have a 8260 device to reproduce the bug easily, so this is just a guess - if you can try installing the kernel at

https://koji.fedoraproject.org/koji/taskinfo?taskID=27279792 

that includes the above patch, and see if something changes, it would be great.
thank you for collaborating,
regards
-- 
davide

[*] I might be wrong, it happens many times. If testing the above patch fails, before searching for another patch on linux-wireless, I will ask you to test wpa_supplicant rebuilt from latest w1.fi repository.

Created attachment 1448582 [details]
Journalctl output with test kernel

(In reply to kwkaess from comment #7)
> Created attachment 1448582 [details]
> Journalctl output with test kernel

More of the same, it seems. Still not working.

Created attachment 1448934 [details]
journalctl -f + lshw -c net from 2 laptops

journalctl -f output + lshw -c net from two laptops effected

(In reply to Brendan Shephard from comment #9)
> Created attachment 1448934 [details]
> journalctl -f + lshw -c net from 2 laptops
> 
> journalctl -f output + lshw -c net from two laptops effected

Still having the same issue with the updated Kernel.

(In reply to Davide Caratti from comment #6)
> (In reply to Brendan Shephard from comment #5)
> > (In reply to Davide Caratti from comment #3)
> > > (In reply to kwkaess from comment #2)
> > >  
> > > > Editing the connection as you instructed ~did~ work, but since my NIC does
> > > > support PMF, I don't consider this an ideal solution.
> > > 
> > > sure, that's not a solution. But with a working wireless connection you are
> > > eligible for testing another kernel / another linux-firmware (and in the
> > > meanwhile we can try debugging more in deep).
> > > 
> > > Is that ok for you?
> > 
> > Hi Davide,
> > 
> > We have about 5 people experiencing this issue now as well. Is there any
> > additional debugging that is required? Or are we pretty happy we know what
> > the issue is here?
> 
> all of you 5 are using Intel 8260, and all of you are seeing failures when
> the IGTK is installed (like in comment #1) ?
>  
> > Happy to help, in the mean-time we'll keep an eye out for a package coming
> > into updates-testing addressing this issue.
> 
> if all of you are seeing the same error message as the one in comment #1,
> then most probably the fix is in the kernel, not in the wpa_supplicant
> package [*].
>

hello,

I reproduced the problem (set_key() getting -EINVAL from the kernel when the IGTK is configured) on my Intel 8260, using f28 kernel (4.16.14-300.fc28.x86_64), wpa_supplicant compiled from upstream git repo, and the following configuration:

ap_scan=1
network={
        ssid="testPMF24"
        #psk="12345678"
        ieee80211w=1
        key_mgmt=WPA-PSK WPA-PSK-SHA256
        pairwise=CCMP
        group=CCMP
        group_mgmt=AES-128-CMAC
}

Forcing swcrypto off [*] (so that iwl_mvm_mac_set_key() installs the key in hardware and returns 0), lets me associate correctly to the BSS that is advertiing PMF. 

Does this happen to you also?

(in the meanwhile I can investigate more what is failing in the mac80211 module)

thank you for testing,

-- 
davide

[*] to turn swcrypto off:
- edit /etc/modprobe.d/nohwcrypt.conf and change

options iwlwifi swcrypto=1

into 

options iwlwifi swcrypto=1

than save the file; as root, run 

# rmmod iwlmvm iwlwifi
# modprobe iwlwifi

then, restart wpa_supplicant and retest connectivity

(In reply to Davide Caratti from comment #11)
> [*] to turn swcrypto off:
> - edit /etc/modprobe.d/nohwcrypt.conf and change
> 
> options iwlwifi swcrypto=1
> 
> into 
> 
> options iwlwifi swcrypto=0
> 

(setting needinfo on reporters, and fixing a typo in comment #11)

(In reply to kwkaess from comment #2)
> Relevant output of $ lshw -c net
> 
>   *-network
>        description: Wireless interface
>        product: Wireless 8260
>        vendor: Intel Corporation
>        physical id: 0
>        bus info: pci@0000:02:00.0
>        logical name: wlp2s0
>        version: 3a
>        serial: <numbers>
>        width: 64 bits
>        clock: 33MHz
>        capabilities: pm msi pciexpress bus_master cap_list ethernet physical
> wireless
>        configuration: broadcast=yes driver=iwlwifi
> driverversion=4.16.11-300.fc28.x86_64 firmware=34.0.1 ip=192.168.0.32
> latency=0 link=yes multicast=yes wireless=IEEE 802.11
>        resources: irq:41 memory:f0c00000-f0c01fff

...

> Editing the connection as you instructed ~did~ work, but since my NIC does
> support PMF, I don't consider this an ideal solution.

after some more investigation: the 8260 supports PMF only when swcrypto is turned off; the relevant lines of code in the iwlfifi driver are at [1].
As you can read, the driver does not enable ciphers required to verify the management frame integrity, unless swcrypto is turned off.

You can verify this with:

# iw phy0 info > /tmp/swcrypto.txt
# sed -i 's/options iwlwifi swcrypto=1/options iwlwifi swcrypto=0/1' /etc/modprobe.d/nohwcrypt.conf
# rmmod iwlmvm iwlwifi mac80211 cfg8021
# modprobe iwlwifi
# iw phy0 info > /tmp/noswcrypto.txt
# diff -u /tmp/swcrypto.txt /tmp/noswcrypto.txt

So, users of iwlwifi can connect to WiFi using one of the following workarounds:
- use swcrypto and disallow PMF
- don't use swcrypto and allow PMF.

I think that we can improve the usability of all this, avoiding situations where NM configures something that is not supported by the hardware: I will discuss this with Beniamino, eventually open a new BZ for that, and keep you in CC. Since there is no activity to be done on wpa_supplicant package, I'm for closing this bz as NOTABUG.

thank you,
-- 
davide

[1]: https://elixir.bootlin.com/linux/v4.18-rc1/source/drivers/net/wireless/intel/iwlwifi/mvm/mac80211.c#L486

Thanks Davide,

Apologies for the lack of action on my behalf last week; been a bit snowed under. Appreciate your investigation into this issue. I agree that we can probably improve the user experience here by making sure we don't have both options when it's not supported by the hardware. 

I'll follow along with the new BZ once it's opened. 

Thanks again,

Hi,

I have a patch for NM to disable optional PMF when the wireless
interface does not support needed ciphers and thus to avoid the
connection failure.

But before I open a bugzilla for it, I'd like to know if I'm solving
the right problem. Davide has hardware crypto disabled by a
configuration file in /etc/modprobe.d. Do all reporters have the same?
kwkaess, Brendan or anyone experiencing this: can you please attach
the output of 'iw phy0 info' and 'grep iwlwifi /etc/modprobe.d/*' ?
Thanks!

Created attachment 1453207 [details]
[PATCH] wifi: disable optional PMF when the NIC doesn't support BIP ciphers

In the meantime I'm attaching the NM patch so that I don't forget it.

Created attachment 1455411 [details]
Output from iw phy0 info

I don't have any custom kernel parameters set in /etc/sysctl.d

*** Bug 1595319 has been marked as a duplicate of this bug. ***

FWIW my Dell XPS 15z laptop didn't connect to any WiFi post its fc28 upgrade
(from fc27 via distro-sync).   
Tried 3 different AP's that all worked prior to fc28.

wpa_supplicant-2.6-16.fc28.x86_64

What worked for me was similar to one (typo'd) suggestion above,
after setting pmf to 0 and 1 didn't make any difference...

Created /etc/modprobe.d/nohwcrypt.conf with 'options iwlwifi swcrypto=1' in it,
and restarted wpa_supplicant and rmmod'd and modprobed the iwl drivers.

$ lsmod|grep iwl
iwldvm                270336  0
mac80211              909312  1 iwldvm
iwlwifi               262144  1 iwldvm
cfg80211              770048  3 iwldvm,iwlwifi,mac80211

Haven't tried swcrypto=0 as was magically working as-is.

Even removing that file and restarting wpa_supplicant and even rebooting; 
I can still connect to stuff, so something magically changed...


# lshw -c net
  *-network                 
       description: Wireless interface
       product: Centrino Advanced-N 6230 [Rainbow Peak]
       vendor: Intel Corporation
       physical id: 0
       bus info: pci@0000:03:00.0
       logical name: wlp3s0
       version: 34
       serial: 88:53:2e:7b:0a:4d
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress bus_master cap_list ethernet physical wireless
       configuration: broadcast=yes driver=iwlwifi driverversion=4.17.3-200.fc28.x86_64 firmware=18.168.6.1 ip=192.168.1.102 latency=0 link=yes multicast=yes wireless=IEEE 802.11
       resources: irq:35 memory:f1b00000-f1b01fff

FWIW, the 'swcrypto' parameter default is off. If some folks have been turning it on then that is a problem.

My Intel 7260 wifi, using DEFAULTS, is connecting fine with APs that have MFP off and APs that have MFP optional.

I know that this bug is closed but the problem still exists on a current Fedora 29 with yet another WiFi chip:

# lshw -c net
*-network
       description: Wireless interface
       product: Centrino Advanced-N 6205 [Taylor Peak]
       vendor: Intel Corporation
       physical id: 0
       bus info: pci@0000:03:00.0
       logical name: wlp3s0
       version: 34
       serial: 60:67:20:b7:97:6c
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress bus_master cap_list ethernet physical wireless
       configuration: broadcast=yes driver=iwlwifi driverversion=4.19.9-300.fc29.x86_64 firmware=18.168.6.1 ip=192.168.2.146 latency=0 link=yes multicast=yes wireless=IEEE 802.11
       resources: irq:30 memory:f1c00000-f1c01fff

This is to confirm that these commands (provided above) issued as root solved the issue (I don't know where this should go to fix the bug systemwise instead of manually):
# nano /etc/modprobe.d/nohwcrypt.conf
// put this line into the file:
// options iwlwifi swcrypto=0

# rmmod iwldvm iwlwifi
# modprobe iwlwifi
# systemctl restart wpa_supplicant

(In reply to kartoffelsalat from comment #21)
> I know that this bug is closed but the problem still exists on a current
> Fedora 29 with yet another WiFi chip:
> 
> # lshw -c net
> *-network
>        description: Wireless interface
>        product: Centrino Advanced-N 6205 [Taylor Peak]
>        vendor: Intel Corporation
>        physical id: 0
>        bus info: pci@0000:03:00.0
>        logical name: wlp3s0
>        version: 34
>        serial: 60:67:20:b7:97:6c
>        width: 64 bits
>        clock: 33MHz
>        capabilities: pm msi pciexpress bus_master cap_list ethernet physical
> wireless
>        configuration: broadcast=yes driver=iwlwifi
> driverversion=4.19.9-300.fc29.x86_64 firmware=18.168.6.1 ip=192.168.2.146
> latency=0 link=yes multicast=yes wireless=IEEE 802.11
>        resources: irq:30 memory:f1c00000-f1c01fff
> 
> This is to confirm that these commands (provided above) issued as root
> solved the issue (I don't know where this should go to fix the bug
> systemwise instead of manually):
> # nano /etc/modprobe.d/nohwcrypt.conf
> // put this line into the file:
> // options iwlwifi swcrypto=0

hello,

the above setting was intended just for analyzing the problem better. I suggest you to undo it, and try the following items:
1) check the system log, e.g with the command

$ sudo journalctl -u wpa_supplicant -n5000  

and see if you find errors similar to the one in comment #1:

May 24 23:07:11 localhost wpa_supplicant[2828]: wlp2s0: RSN: Failed to configure IGTK

2) try disabling PMF in network manager (you can find instructions at https://fedoramagazine.org/troubleshoot-pmf-f28/)

please let me know if that helped or not.
thanks!
-- 
davide

> please let me know if that helped or not.

Sorry, this issue concerned the laptop of a friend. It's a bit difficult to borrow the device to try your suggestions, given that the WiFi is working properly right now.

The issues I experienced seemed to have vanished in a kernel update about 2 weeks ago.  4.20.5 or thereabouts.

*** Bug 1655535 has been marked as a duplicate of this bug. ***

Beniamino Galvani, did you open the new BZ as mentioned in comment 15/comment 16?

BTW, this bug has been marked as a dupicate of bug 1595319 and vice versa.

(In reply to Russell Odom from comment #26)
> Beniamino Galvani, did you open the new BZ as mentioned in comment
> 15/comment 16?

I didn't because a better fix was included in NM [1]. With that commit, the logic of deciding whether the station supports optional PMF is delegated to wpa_supplicant. The fix is available in NM 1.16 and later.

[1] https://gitlab.freedesktop.org/NetworkManager/NetworkManager/commit/a9ab50efb10dfb50cfe897c58afa300f8b07f6ba

OK, thanks, that explains it. NetworkManager-1.12.6-5.fc29.x86_64 installed here.