Bug 1656157

Summary: NetworkManager 1.12.6 breaks brcmfmac43241b4-sdio wifi (WPA2)
Product: [Fedora] Fedora Reporter: Hans de Goede <hdegoede>
Component: NetworkManagerAssignee: Beniamino Galvani <bgalvani>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 29CC: bgalvani, dcbw, fgiudici, john.j5live, lkundrak, mclasen, rhughes, rstrode, sandmann, thaller
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: NetworkManager-1.12.6-2.fc29 NetworkManager-1.12.6-3.fc29 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-12-15 03:18:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
NetworkManager.service log
none
wpa_supplicant.service log
none
NetworkManager.service log
none
wpa_supplicant.service log
none
[PATCH] device: always take device down when changing MAC for wifi devices
none
[PATCH v2] device: always take device down when changing MAC for wifi devices none

Description Hans de Goede 2018-12-04 20:52:27 UTC
Created attachment 1511466 [details]
NetworkManager.service log

After updating from NetworkManager 1.12.4 to 1.12.6 the wifi on a Lenovo Miix 2 8, using a brcmfmac43241b4-sdio nic, no longer connects.

Instead it errors out that credentials are required even if a passwd is specified on the nmcli command line.

Downgrading to 1.12.4 restores wifi connectivity.

I'm attaching journalctl -b0 -u Networkmanager.service and wpa_supplicant.service output, let me know if you need any more logs.

Note there are a lot of mywifi connections configured in NM because this is from an USB SSD which I used to test various devices with the latest Fedora, so for each new device/MAC there is a new connection.

Comment 1 Hans de Goede 2018-12-04 20:53:13 UTC
Created attachment 1511467 [details]
wpa_supplicant.service log

Comment 2 Hans de Goede 2018-12-04 20:54:27 UTC
p.s. I've already tried upgrading wpa_supplicant to 2.6-20.fc29.x86_64 this does not help.

Comment 3 Beniamino Galvani 2018-12-04 21:34:11 UTC
Hi,

can you please try adding this at the end of /etc/NetworkManager/NetworkManager.conf:

[device-mac-addr-change-wifi]
wifi.scan-rand-mac-address=no

and restart the NetworkManager service?

I suspect the regression is caused by commit

https://gitlab.freedesktop.org/NetworkManager/NetworkManager/commit/b81648fc6c272e0f2bf3116168844737eecfa354

because wpa_supplicant assumes that the MAC address can only change when the interface is down. To verify this can you please capture wpa_supplicant logs without the above workaround after adding the '-dd' option OTHER_ARGS in /etc/sysconfig/wpa_supplicant (and restarting the service)? Also, NetworkManager logs with trace level would be helpful (set level=trace in the [logging] service of /etc/NetworkManager/NetworkManager.conf). Thanks.

Comment 4 Hans de Goede 2018-12-06 14:04:37 UTC
Created attachment 1512145 [details]
NetworkManager.service log

Comment 5 Hans de Goede 2018-12-06 14:05:08 UTC
Created attachment 1512146 [details]
wpa_supplicant.service log

Comment 6 Hans de Goede 2018-12-06 14:07:00 UTC
Created attachment 1512148 [details]
NetworkManager.service log

Comment 7 Hans de Goede 2018-12-06 14:07:26 UTC
Created attachment 1512149 [details]
wpa_supplicant.service log

Comment 8 Hans de Goede 2018-12-06 14:09:44 UTC
Hi,

I've attached new logs of a fresh boot with the new NM + connect failure using the requested debug options.

After gathering these logs I've added:

[device-mac-addr-change-wifi]
wifi.scan-rand-mac-address=no

And restarted NM, I can confirm that this fixes / works around the problem.

Regards,

Hans

Comment 9 Beniamino Galvani 2018-12-06 16:10:53 UTC
Created attachment 1512179 [details]
[PATCH] device: always take device down when changing MAC for wifi devices

Comment 10 Thomas Haller 2018-12-06 17:44:04 UTC
(In reply to Beniamino Galvani from comment #9)
> Created attachment 1512179 [details]
> [PATCH] device: always take device down when changing MAC for wifi devices

was_taken_down is left uninitalized.


Also, while a suitable workaround, this is ultimately a bug in supplicant. It should be at least reported, preferably fixed. Also, it may not be a suitable workaround for macsec or 802-1x.

Comment 11 Beniamino Galvani 2018-12-10 10:31:55 UTC
Created attachment 1513004 [details]
[PATCH v2] device: always take device down when changing MAC for wifi  devices

(In reply to Thomas Haller from comment #10)
> Also, while a suitable workaround, this is ultimately a bug in supplicant.
> It should be at least reported, preferably fixed.

I think the patch should be applied to NM in any case; even if we try to submit wpa_supplicant patches to improve this (which is not a trivial task), they will probably take long time (months?) to appear in a stable release. If they are accepted, which is not granted.

> Also, it may not be a suitable workaround for macsec or 802-1x.

For MACsec and 802.1X Ethernet we create the supplicant interface only when the connection is being activated and we do it in stage2, after the MAC was changed in stage1. So there isn't any MAC change while wpa_supplicant is managing the interface.

For Wi-Fi it is different because we need to scan and so the supplicant interface is created when the device is managed. Then we change the MAC on activation in stage1.

Comment 12 Thomas Haller 2018-12-10 11:02:09 UTC
v2 lgtm

Comment 14 Fedora Update System 2018-12-11 13:21:08 UTC
NetworkManager-1.12.6-3.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2018-3edec78f47

Comment 15 Fedora Update System 2018-12-12 03:33:20 UTC
NetworkManager-1.12.6-3.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-3edec78f47

Comment 16 Hans de Goede 2018-12-13 13:19:52 UTC
I can confirm that the 1.12.6-3 build fixes this (wifi no longer working on a device with a brcmfmac43241b4-sdio nic).

Comment 17 Fedora Update System 2018-12-15 03:18:39 UTC
NetworkManager-1.12.6-3.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.