Bug 2072070 - Connection to wireless network fails without explanation when other end does not support secure renegotiation [NEEDINFO]
Summary: Connection to wireless network fails without explanation when other end does ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: wpa_supplicant
Version: 36
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Lubomir Rintel
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedBlocker
: 2076754 (view as bug list)
Depends On:
Blocks: F36FinalBlocker
TreeView+ depends on / blocked
 
Reported: 2022-04-05 14:20 UTC by Reese Armstrong (reesericci)
Modified: 2022-05-19 07:49 UTC (History)
33 users (show)

Fixed In Version: wpa_supplicant-2.10-4.fc36
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-05-05 18:37:18 UTC
Type: Bug
kickdown: needinfo? (ralston)
kickdown: needinfo? (me)
kickdown: needinfo? (neilanthonydesignbuild)
kickdown: needinfo? (michel)


Attachments (Terms of Use)

Description Reese Armstrong (reesericci) 2022-04-05 14:20:53 UTC
Description of problem:

F36 (Silverblue) refuses to connect to the WPA2 Enterprise network at school. It tries to connect for a minute or so and then just gives up. The network uses PEAP + MSChapv2 with no CA Certificate. When I roll back to F35, the network connects fine.

Version-Release number of selected component (if applicable):

NetworkManager version 1.36.4-1.fc36

GNOME version 42.0

OS Name Fedora Linux 36.20220404.n.0 (Silverblue Prerelease)


How reproducible:

Yes?

Steps to Reproduce:

1. Try to connect to a WPA2 Enterprise Wi-Fi network with PEAP + MSChapv2 no CA certificate. (F36 Silverblue GNOME)

Actual results:

Tries to connect for a minute, and then fails silently. Sometimes it pops up a prompt to enter my username and password again.


Expected results:

Connected to network properly

Additional info:

Comment 1 David Koppelman 2022-04-05 14:48:30 UTC
I too am having a problem connecting via wpa2 Enterprise (otherwise
wifi works normally). This is on a system that had been upgraded from
f35 to f36 on 2022-02-19 13:27. I encountered the wpa problem after a
2022-02-23 10:44 upgrade, and I recall that the wpa connection was working
fine before then on f36.

The relevant log entries are:

    Feb 23 10:58:47 dmk-laptop-21 wpa_supplicant[1404]: wlp147s0: CTRL-EVENT-EAP-METHOD EAP vendor 0 method 25 (PEAP) selected
    Feb 23 10:58:47 dmk-laptop-21 wpa_supplicant[1404]: SSL: SSL3 alert: write (local SSL3 detected an error):fatal:handshake failure
    Feb 23 10:58:47 dmk-laptop-21 wpa_supplicant[1404]: OpenSSL: openssl_handshake - SSL_connect error:0A000152:SSL routines::unsafe legacy renegotiation disabled
    Feb 23 10:58:48 dmk-laptop-21 wpa_supplicant[1404]: wlp147s0: CTRL-EVENT-EAP-FAILURE EAP authentication failed

The same issue has been reported here:
https://ask.fedoraproject.org/t/cannot-connect-to-wpa2-enterprise-university-wifi-on-fedora-36/20288

Comment 2 Thomas Haller 2022-04-05 14:56:35 UTC
does `update-crypto-policies --set legacy` and reboot make it work?

It seems it's related to disabled crypto (from openssl, used by wpa_supplicant).

Degrading the entire system to "legacy" however is not the right solution and I don't recommend that.
(I don't know what the right solution is).

Comment 3 David Koppelman 2022-04-05 15:05:37 UTC
(In reply to Thomas Haller from comment #2)
> does `update-crypto-policies --set legacy` and reboot make it work?
> 
> It seems it's related to disabled crypto (from openssl, used by
> wpa_supplicant).
> 
> Degrading the entire system to "legacy" however is not the right solution
> and I don't recommend that.
> (I don't know what the right solution is).

I'll give it a try.

If indeed the problem is due to my institution's AP using an unsafe protocol, the right solution is to prepare instructions for collecting information that can be used to file a ticket to those managing the wireless access point. I'd like to file a ticket with a subject line like: "Please upgrade AP software from Cisco version X to at least Y which includes encrypted handshaking. Currently my OS considers the AP too unsafe to connect." Is it possible to get such information?

Comment 4 David Koppelman 2022-04-05 15:33:33 UTC
Same problem with legacy settings after a reboot:

[root@dmk-laptop-21 koppel]# update-crypto-policies --show
LEGACY


Apr 05 10:22:45 dmk-laptop-21 wpa_supplicant[1381]: wlp147s0: Associated with 00:42:68:62:55:6a
Apr 05 10:22:45 dmk-laptop-21 wpa_supplicant[1381]: wlp147s0: CTRL-EVENT-EAP-STARTED EAP authentication started
Apr 05 10:22:45 dmk-laptop-21 wpa_supplicant[1381]: wlp147s0: CTRL-EVENT-SUBNET-STATUS-UPDATE status=0
Apr 05 10:22:45 dmk-laptop-21 NetworkManager[1285]: <info>  [1649172165.4588] device (wlp147s0): supplicant interface state: associating -> associated
Apr 05 10:22:45 dmk-laptop-21 NetworkManager[1285]: <info>  [1649172165.4589] device (p2p-dev-wlp147s0): supplicant management interface state: associating -> associated
Apr 05 10:22:45 dmk-laptop-21 wpa_supplicant[1381]: wlp147s0: CTRL-EVENT-EAP-PROPOSED-METHOD vendor=0 method=26 -> NAK
Apr 05 10:22:45 dmk-laptop-21 wpa_supplicant[1381]: wlp147s0: CTRL-EVENT-EAP-PROPOSED-METHOD vendor=0 method=25
Apr 05 10:22:45 dmk-laptop-21 wpa_supplicant[1381]: wlp147s0: CTRL-EVENT-EAP-METHOD EAP vendor 0 method 25 (PEAP) selected
Apr 05 10:22:45 dmk-laptop-21 wpa_supplicant[1381]: SSL: SSL3 alert: write (local SSL3 detected an error):fatal:handshake failure
Apr 05 10:22:45 dmk-laptop-21 wpa_supplicant[1381]: OpenSSL: openssl_handshake - SSL_connect error:0A000152:SSL routines::unsafe legacy renegotiation disabled
Apr 05 10:22:45 dmk-laptop-21 kernel: wlp147s0: Limiting TX power to 12 dBm as advertised by 00:42:68:62:55:6a
Apr 05 10:22:46 dmk-laptop-21 wpa_supplicant[1381]: wlp147s0: CTRL-EVENT-EAP-FAILURE EAP authentication failed

Comment 5 Reese Armstrong (reesericci) 2022-04-06 13:46:07 UTC
The legacy crypto policies didn't work for me either.

Comment 6 Reese Armstrong (reesericci) 2022-04-07 12:17:04 UTC
Is there a chance that this can be a blocker for F36 Final? This is going to break a lot of people's systems at work/school.

Comment 7 David Koppelman 2022-04-07 14:49:38 UTC
(In reply to Reese Armstrong (reesericci) from comment #6)
> Is there a chance that this can be a blocker for F36 Final? This is going to
> break a lot of people's systems at work/school.

OP, here's how to propose it:

https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process

This may be the relevant criterion, though it says nothing about specific methods of connecting to a network, not to mention, if a particular method is common enough to be considered a blocker:
https://fedoraproject.org/wiki/Basic_Release_Criteria#Remote_package_sources

Comment 8 Reese Armstrong (reesericci) 2022-04-08 01:19:35 UTC
OK, I just requested it in Bugzilla as a Final Blocker, we'll see if it gets some attention that way.

Comment 9 Thomas Haller 2022-04-08 06:53:18 UTC
in any case, this does not seem to be a NetworkManager issue. Either wpa_supplicant or openssl. Reassigning.

Comment 10 Thomas Haller 2022-04-08 06:55:23 UTC
as this rhbz links to the following question, did you try that suggestion? Did it work?
https://ask.fedoraproject.org/t/cannot-connect-to-wpa2-enterprise-university-wifi-on-fedora-36/20288/5

Comment 11 Kamil Páral 2022-04-08 11:33:00 UTC
(In reply to Reese Armstrong (reesericci) from comment #6)
> Is there a chance that this can be a blocker for F36 Final? This is going to
> break a lot of people's systems at work/school.

It can't be a blocker because Silverblue is not a blocking deliverable:
https://docs.fedoraproject.org/en-US/releases/f36/blocking/

It can be a freeze exception, though. Changing the tags.

Comment 12 Kamil Páral 2022-04-08 12:29:03 UTC
Actually, the Ask link in comment 1 talks about both Silverblue and Workstation. Putting the blocker nomination back.

Comment 13 Reese Armstrong (reesericci) 2022-04-08 12:53:25 UTC
(In reply to Thomas Haller from comment #10)
> as this rhbz links to the following question, did you try that suggestion?
> Did it work?
> https://ask.fedoraproject.org/t/cannot-connect-to-wpa2-enterprise-university-
> wifi-on-fedora-36/20288/5

I did take a look at the post, but haven't been able to try it yet. I'll try it when I get to school in an hour. Either way, I shouldn't have to do anything in order to get this to work, as I did in F35 (Workstation & Silverblue).

Comment 14 David Koppelman 2022-04-08 15:59:48 UTC
(In reply to Thomas Haller from comment #10)
> as this rhbz links to the following question, did you try that suggestion?
> Did it work?
> https://ask.fedoraproject.org/t/cannot-connect-to-wpa2-enterprise-university-
> wifi-on-fedora-36/20288/5

I can now connect (but I don't think the issue has been resolved). As suggested in that ask.fedoraproject.org post I edited /etc/pki/tls/openssl.cnf and I added the Options line so that the crypto_policy section appears as follows:

    [ crypto_policy ]

    Options = UnsafeLegacyRenegotiation
    .include = /etc/crypto-policies/back-ends/opensslcnf.config

That worked around the problem for me. That is, I can now connect. I would still like this written up somewhere, perhaps in release notes, so that those encountering the problem can verify that it's the handshake issue and to provide the kind of information needed to file a ticket. (See Comment 3).

Comment 15 František Zatloukal 2022-04-11 18:17:18 UTC
Discussed during the 2022-04-11 blocker review meeting: [1]

The decision to classify this bug as an RejectedBlocker was made:

"This is a feature of openssl and not a bug. We will mark it in commonbugs for reference."

[1] https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2022-04-11/f36-blocker-review.2022-04-11-16.00.log.txt

Comment 16 Reese Armstrong (reesericci) 2022-04-11 19:06:23 UTC
Excuse me, what? Not being able to connect to WPA2 Enterprise networks is a feature? That is ridiculous.

Comment 17 František Zatloukal 2022-04-11 21:22:12 UTC
Discussed during the 2022-04-11 blocker review meeting: [1]

The decision to classify this bug as an RejectedFreezeException was made:

"The bug in question is actually expected behavior in openssl. We reject it as an FE on this basis."

[1] https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2022-04-11/f36-blocker-review.2022-04-11-16.00.log.txt

Comment 18 Reese Armstrong (reesericci) 2022-04-12 02:45:08 UTC
So what should be done about this? The UX is terrible right now, and how do we convey this to the end user? Putting it in release notes and calling it a day is not a solution to the issue at hand.

Comment 19 Scott Dowdle 2022-04-12 17:00:31 UTC
Putting it in the release notes would probably be the best way if the "fix" isn't going to be added to a package update.

Comment 20 Adam Williamson 2022-04-18 22:18:03 UTC
"Excuse me, what? Not being able to connect to WPA2 Enterprise networks is a feature? That is ridiculous."

No, that's not an accurate description of the issue AFAIK. The problem will happen only with networks that do not support secure renegotiation. This is to do with the design or configuration of the router (or whatever is handling the other end of the connection). See the "SECURE RENEGOTIATION" section of https://www.openssl.org/docs/man3.0/man3/SSL_clear_options.html for more details.

I agree that the user not getting an explanation of why this failed is a problem; retitling the bug to reflect that. wpa_supplicant clearly *does* report the problem, but something further up the chain swallows it. Moving to NetworkManager for now at least - I'm not clear whether NetworkManager is not reporting the error to GNOME, or GNOME is getting some indication of the error from NM but not passing it on to the user.

Comment 21 Reese Armstrong (reesericci) 2022-04-19 11:50:17 UTC
That changes things. Thanks for clearing that up! I will file a complaint with my network admin about this.

--reese

Comment 22 James Ralston 2022-04-27 22:39:14 UTC
(In reply to Adam Williamson from comment #20)
> "Excuse me, what? Not being able to connect to WPA2 Enterprise networks is a
> feature? That is ridiculous."
> 
> No, that's not an accurate description of the issue AFAIK.

I respectfully disagree. This *specific use case* is a bug, not a feature, and this should absolutely be a blocker for F36:

1. The real-world security implications of permitting insecure renegotiation during the PEAP TLS session are negligible. There is zero risk to the client; an MitM attacker cannot obtain user credentials or fake the authentication.

2. We also encountered this at our organization, with RHEL9 beta. Despite running a modern Wi-Fi infrastructure with modern Wi-Fi equipment, our network team reports that the Wi-Fi equipment provides no mechanism to enable RFC5746 secure renegotiation on the EAP server. This is likely because until OpenSSL 3.0, literally *nothing* cared about whether any EAP server anywhere supported RFC5746. Wi-Fi hardware vendors simply *do not care* about this. As such, getting RFC5746 support will not be a quick fix; this will take literally years/decades.

3. Working around this is horrendous: it requires modifying the wpa_supplicant systemd unit file to point to a different openssl.cnf file, then manually creating that openssl.cnf file and adding the “UnsafeLegacyRenegotiation” configuration option. See: https://bugs.launchpad.net/ubuntu/+source/wpa/+bug/1958267/comments/22

Fedora 36 *must* ship with the work-around (point #3 above) enabled by default.

Long-term, there needs to be an option in the “Wi-Fi Security” tab in the NetworkManager configuration to enable unsafe TLS renegotiation during PEAP authentication, and both NetworkManager and wpa_supplicant must coordinate to ensure that the user understands what is happening (and how to resolve it) when PEAP authentication fails because the PEAP server does not support RFC5746. For more information, see: https://bugzilla.redhat.com/show_bug.cgi?id=2077973#c6

Comment 23 Adam Williamson 2022-04-27 23:31:34 UTC
A downstream bug in Fedora does not seem like the place to debate upstream openssl security policy choices, to me.

Your description of the workaround as "horrendous" does not seem accurate at least on Fedora, per comment #14 just editing a single file is sufficient.

Comment 24 James Ralston 2022-04-28 06:56:35 UTC
Ok, I’ve dug into this some more.

First: the UnsafeLegacyRenegotiation option (corresponding to SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION) is the wrong option. That will enable unsafe legacy renegotiation in *both* client and server contexts. That’s bad. The correct option to use here is UnsafeLegacyServerConnect (corresponding to SSL_OP_LEGACY_SERVER_CONNECT), which allows only TLS clients to connect to TLS servers that permit unsafe legacy renegotiation, while still causing server instances to reject a TLS client that does not support RFC5746 secure renegotiation.

(In reply to Adam Williamson from comment #23)
> A downstream bug in Fedora does not seem like the place to debate upstream
> openssl security policy choices, to me.

I respectfully disagree.

Quoting the migration guide <https://www.openssl.org/docs/man3.0/man7/migration_guide.html#TLS-Changes>:

> Secure renegotiation is now required by default for TLS connections
> 
> Support for RFC 5746 secure renegotiation is now required by default
> for SSL or TLS connections to succeed. Applications that require the
> ability to connect to legacy peers will need to explicitly set
> SSL_OP_LEGACY_SERVER_CONNECT. Accordingly,
> SSL_OP_LEGACY_SERVER_CONNECT is no longer set as part of SSL_OP_ALL.

There have been independent reports of Wi-Fi PEAP breakage for Fedora 36 beta, RHEL9 beta, and other Linux distributions that have started shipping OpenSSL 3.0. This is a strong indication that lack of RFC5746 support in enterprise vendor Wi-Fi hardware is a widespread issue.

Thus, wpa_supplicant is an application that requires the ability to connect to legacy TLS servers, and per the OpenSSL migration, it is not just appropriate but necessary for wpa_supplicant to set the SSL_OP_LEGACY_SERVER_CONNECT option.

wpa_supplicant’s failure to set SSL_OP_LEGACY_SERVER_CONNECT is a bug. It is a bug that predates OpenSSL 3.0, but was occulted because before OpenSSL 3.0, SSL_OP_ALL included SSL_OP_LEGACY_SERVER_CONNECT and thus masked the bug. So it’s perfectly understandable why this bug was not caught until now. But it is a bug nonetheless.

> Your description of the workaround as "horrendous" does not seem accurate at
> least on Fedora, per comment #14 just editing a single file is sufficient.

Again, I respectfully disagree.

First: the edit suggested in comment #14 is not just completely inappropriate but *dangerous*: setting “Options = UnsafeLegacyRenegotiation” in the default (system-wide) openssl.cnf file will permit unsafe renegotiations, for *all* applications linked against OpenSSL, in *both* client and server contexts. *THIS WILL EFFECTIVELY RE-ENABLE CVE-2009-3555 SYSTEM-WIDE*. I know David (the comment author) didn’t intend that comment to be bad advice, and I’m not trying to throw him under the bus here, but bad advice it certainly is.

Even the safer “Options = UnsafeLegacyServerConnect” (which permits only TLS clients to connect to TLS servers that don’t support RFC5746, while still defaulting TLS servers to require clients to support RFC5746) should not be placed in the system-wide openssl.cnf file, as that re-enables CVE-2009-3555 for all OpenSSL TLS clients on the system. This effectively defeats the security policy choice made in OpenSSL 3.0 to no longer set SSL_OP_LEGACY_SERVER_CONNECT as part of SSL_OP_ALL (because enough TLS server implementations now support RFC5746 that it is appropriate to place to onus of setting SSL_OP_LEGACY_SERVER_CONNECT onto the individual applications that must communication with legacy servers and thus specifically require it).

To properly work around this wpa_supplicant bug without re-enabling CVE-2009-3555 system-wide, the user must do the following:

$ cat 1>>/etc/sysconfig/wpa_supplicant <<EOF

# See /etc/wpa_supplicant/openssl.cnf for the explanation why using a custom
# openssl.cnf file for wpa_supplicant (which is what setting OPENSSL_CONF here
# does) is necessary.

OPENSSL_CONF="/etc/wpa_supplicant/openssl.cnf"
EOF
$ cp -p /etc/pki/tls/openssl.cnf /etc/wpa_supplicant/openssl.cnf
$ cd /etc/wpa_supplicant
$ patch -p0 <<EOF
--- openssl.cnf.old	2022-03-18 12:49:28.000000000 -0400
+++ openssl.cnf	2022-04-28 01:08:10.260379609 -0400
@@ -36,6 +36,24 @@
 
 [ crypto_policy ]
 
+# Starting with OpenSSL 3.0, OpenSSL no longer includes
+# SSL_OP_LEGACY_SERVER_CONNECT in SSL_OP_ALL, as most TLS servers now support
+# RFC5746 TLS secure renegotiation.  TLS client applications that specifically
+# need to connect to TLS servers that do not support RFC5746 TLS secure
+# renegotiation should (and must) explicitly set SSL_OP_LEGACY_SERVER_CONNECT.
+#
+# Wi-Fi PEAP authentication uses TLS, and a nontrivial number of enterprise
+# Wi-Fi implementations that perform PEAP authentication fail to implement
+# RFC5746 TLS secure renegotiation.  But wpa_supplicant does not set
+# SSL_OP_LEGACY_SERVER_CONNECT when it performs client PEAP authentication,
+# because the pre-3.0 OpenSSL behavior of including
+# SSL_OP_LEGACY_SERVER_CONNECT in SSL_OP_ALL occulted the need to do so.
+#
+# Until wpa_supplicant properly sets the SSL_OP_LEGACY_SERVER_CONNECT option,
+# we must do so manually.
+
+Options = UnsafeLegacyServerConnect
+
 .include = /etc/crypto-policies/back-ends/opensslcnf.config
 
 [ new_oids ]

EOF
$ restorecon -F /etc/wpa_supplicant/openssl.cnf
$ systemctl try-restart wpa_supplicant

I stand by my “horrendous” comment. It is completely unacceptable to expect a *user* to do this to work around what is fundamentally a bug with wpa_supplicant.

Moreover, once the user forks a custom openssl.cnf file for wpa_supplicant, it’s going to persist forever, across Fedora upgrades, until the user does a clean install, because /etc/sysconfig/wpa_supplicant is flagged as config(noreplace) in the wpa_supplicant rpm package.

But it’s even worse: how is the user supposed to figure out that this is even necessary to do in the first place?

When the wpa_supplicant bug occurs, NetworkManager behaves exactly as if the user entered an incorrect password. The only clue what is really happening (and where the true problem lies) is the cryptic “SSL_connect error:0A000152:SSL routines::unsafe legacy renegotiation disabled” message that is logged to the system log—which only system administrators can see. From there, it’s off to web searches to try to find what that error means and how to resolve / work around the problem.

If the user is lucky, they’ll stumble across an explanation of how to work around the bug in wpa_supplicant without re-enabling CVE-2009-3555 system-wide (a la comment #14). If the user is unlucky, they’ll render their entire system, including OpenSSL TLS servers, vulnerable to CVE-2009-3555.

The decision to reject this bug as a blocker bug for Fedora 36 was made without a complete understanding of precisely what the bug is. Breaking the ability to connect to PEAP Wi-Fi networks is *NOT* a feature of OpenSSL 3.0, as the OpenSSL 3.0 migration guide is painfully explicit that it is the responsibility of applications that need to connect to legacy TLS servers to set SSL_OP_LEGACY_SERVER_CONNECT. Rather, this is a long-standing bug with wpa_supplicant (assuming SSL_OP_LEGACY_SERVER_CONNECT will be set automatically, instead of manually setting it) that OpenSSL 3.0 simply exposed.

Looking at the wpa_supplicant source code, I strongly suspect this one-line patch will have the same effect as setting “Options = UnsafeLegacyServerConnect” in openssl.cnf:

diff -up wpa_supplicant-2.10/src/crypto/tls_openssl.c.legacy-server-connect wpa_supplicant-2.10/src/crypto/tls_openssl.c
--- wpa_supplicant-2.10/src/crypto/tls_openssl.c.legacy-server-connect  2022-01-16 15:51:29.000000000 -0500
+++ wpa_supplicant-2.10/src/crypto/tls_openssl.c        2022-04-28 02:47:26.863529683 -0400
@@ -1049,6 +1049,16 @@ void * tls_init(const struct tls_config
        SSL_CTX_set_options(ssl, SSL_OP_NO_SSLv2);
        SSL_CTX_set_options(ssl, SSL_OP_NO_SSLv3);
 
+       /* Many enterprise PEAP server implementations (e.g. used in large
+          corporations and universities) do not support RFC5746 secure
+          renegotiation, and starting with OpenSSL 3.0,
+          SSL_OP_LEGACY_SERVER_CONNECT is no longer set as part of SSL_OP_ALL.
+          So until we implement a way to request SSL_OP_LEGACY_SERVER_CONNECT
+          only in EAP peer mode, just set SSL_OP_LEGACY_SERVER_CONNECT
+          globally. */
+
+       SSL_CTX_set_options(ssl, SSL_OP_LEGACY_SERVER_CONNECT);
+
        SSL_CTX_set_mode(ssl, SSL_MODE_AUTO_RETRY);
 
 #ifdef SSL_MODE_NO_AUTO_CHAIN

(WARNING: I have not tested this yet. This is a 1-in-the-morning patch attempt based on a 5-minute perusal of the source code. Please test before using!)

As per my comments on BZ#2077973, long-term, this should be addressed by adding a “permit unsafe legacy TLS renegotiation in PEAP authentication” option to both NetworkManager and wpa_supplicant, with an intelligent error message along the lines of “this Wi-Fi network requires the ‘permit unsafe legacy TLS renegotiation in PEAP authentication’ option to be enabled” communicated back to the user. But until that can be implemented, the simplest way to squash the bug is for wpa_supplicant to unconditionally set SSL_OP_LEGACY_SERVER_CONNECT.

It is not clear to me how to appeal the rejection of a blocker bug nomination, but an appeal is clearly needed here.

Comment 25 Kamil Páral 2022-04-28 07:14:11 UTC
Resetting blocker decision back to nominated as requested in comment 24.

Comment 26 Thomas Haller 2022-04-28 08:12:29 UTC
Comment 24 seems accurate to me. Thanks James for the effort!

The proper solution is mostly in NetworkManager, to improve error reporting and allow to opt-in for unsafe behavior on a per-profile basis. That will be a lot of upstream work (patch welcome).

The workaround in supplicant seems reasonably simple and what we should do for now. I think that supplicant should by default use a different openssl.cnf, which is shipped and maintained by Fedora's supplicant package. That allows the expert user to still modify it, and by default we'd ship a less strict SSL configuration (for now). After all, different users of openssl have different requirements w.r.t. security.

I'd reassign this bug to wpa_supplicant.

Comment 27 Adam Williamson 2022-04-28 15:51:50 UTC
Thanks for the valuable added details, James, you're right that I (and probably others who voted) was not aware of all of those. This will be discussed again at the go/no-go meeting happening in just over an hour in #fedora-meeting on libera.chat or chat.fedoraproject.org , if you want to join in.

Unfortunately we now have to either delay the release again to fix this, or waive it; if your clarification had been available a few days earlier we could've got a fix in :|

Comment 28 Ben Williams 2022-04-28 18:19:25 UTC
tested this with RC1.2 andit connects to the my Uni Network using PEAP and MSChapv2 with no CA Certificate

Comment 29 Ben Cotton 2022-04-28 19:10:30 UTC
In today's Go/No-Go meeting, we accepted this as a blocker as this bug is a conditional violation of all criteria that imply network access and any 'install must work' criteria in cases where an affected WiFi network is used.
https://meetbot.fedoraproject.org/fedora-meeting/2022-04-28/f36-final-go_no_go-meeting.2022-04-28-17.01.log.html#l-258

Comment 30 James Ralston 2022-04-28 19:19:04 UTC
(In reply to Adam Williamson from comment #27)
> Unfortunately we now have to either delay the release again to fix this, or
> waive it; if your clarification had been available a few days earlier we
> could've got a fix in :|

Alas, it took me a little bit to fully wrap my head around what was happening here: I couldn’t understand why SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION was just cropping up now, as OpenSSL disabled unsafe legacy renegotiation (and added that option to prevent it) way back in 1.1.0. Literally nothing about SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION changed in 3.0.

But that was a red herring. What changed in 3.0 was the removal of SSL_OP_LEGACY_SERVER_CONNECT from SSL_OP_ALL. That’s what finally exposed the wpa_supplicant bug.

(In reply to Ben Williams from comment #28)
> tested this with RC1.2 andit connects to the my Uni Network using PEAP and
> MSChapv2 with no CA Certificate

Do you mean you tested with the patch to wpa_supplicant I proposed in comment #24, or you are using an unpatched wpa_supplicant and are unaffected by the bug?

BTW, our network group shared the vendor’s response to their (upstream) ticket with me. The vendor admits their Wi-Fi EAP implementation does not support RFC5746 secure renegotiation. They’ve added it as a feature request for subsequent releases, but feature requests are prioritized based on how many customers request the feature.

(In reply to Thomas Haller from comment #26)
> The proper solution is mostly in NetworkManager, to improve error reporting
> and allow to opt-in for unsafe behavior on a per-profile basis. That will be
> a lot of upstream work (patch welcome).

It’s not just NetworkManager, though. Ideally, wpa_supplicant needs to provide an option so that NetworkManager can ask wpa_supplicant to set SSL_OP_LEGACY_SERVER_CONNECT on a per-connection basis. And if PEAP authentication fails because SSL_OP_LEGACY_SERVER_CONNECT was needed (but not enabled), wpa_supplicant needs to communicate that specific failure case back to NetworkManager, so that NetworkManager can explain (via a dialog pop-up) to the user what the issue is and how to work around it.

If done correctly, this will enlist Linux users to ask the owners of the non-RFC5746-compliant Wi-Fi systems why Linux is muttering about their Wi-Fi being unsafe… which will encourage those owners to submit feature requests to their vendors to support RFC5746.

And let’s be honest: the reason why Wi-Fi vendor’s TLS server implementations have escaped security scrutiny all these years is because they are essentially embedded implementations that are not public. (I will admit that before I encountered this issue, I was unaware that PEAP authentication used TLS under the hood.) So the Eye of Sauron that is https://www.ssllabs.com/ssltest/ (for which failure to support RFC5746 is an automatic “F” grade) has never been able to find them. A little public sham^H^H^H^H^H^H^H^H^H^H^H sunlight will go far here.

(RFC5746 was published in February 2010, ffs. If large Wi-Fi vendors couldn’t even be bothered to implement that, more than a decade later, what other security issues exist in their TLS implementations?)

Comment 31 Adam Williamson 2022-04-29 18:31:40 UTC
So, now this has been accepted as a blocker, we need a build of wpa_supplicant with the workaround in place. Thomas, can you do that? If not, who should we ask? I'm a proven packager so I have the power to do it, but I'm not sure I have a strong enough grasp on all the details here to be sure I'm doing the right thing.

Comment 32 Reese Armstrong (reesericci) 2022-04-29 20:38:55 UTC
I'm very excited to hear that a fix is in the works! Hopefully this doesn't delay the release of F36 too much though...


--reese

Comment 33 Davide Caratti 2022-04-30 17:09:56 UTC
(In reply to Adam Williamson from comment #31)
> So, now this has been accepted as a blocker, we need a build of
> wpa_supplicant with the workaround in place. Thomas, can you do that? If
> not, who should we ask? I'm a proven packager so I have the power to do it,
> but I'm not sure I have a strong enough grasp on all the details here to be
> sure I'm doing the right thing.

givent that the workaround is patch at comment #24, I can manage to do a wpa_supplicant build. Should we try submitting officially the patch to hostapd mailing list, or keep it as non-upstream?

Comment 34 Adam Williamson 2022-04-30 17:19:39 UTC
Well, the author of comment #24 wrote "(WARNING: I have not tested this yet. This is a 1-in-the-morning patch attempt based on a 5-minute perusal of the source code. Please test before using!)", so rather than just slap it in I was hoping we could have a subject matter expert check it over...

I'm not sure whether it's upstream appropriate, that's another thing someone more experienced in the area could say. But for Fedora release timeframe purposes, we can't really wait for upstream approval *before* fixing it downstream, we need to do a downstream build ASAP whether or not we submit it upstream at the same time.

Thanks!

Comment 35 Davide Caratti 2022-04-30 17:55:44 UTC
(In reply to Adam Williamson from comment #34)
> Well, the author of comment #24 wrote "(WARNING: I have not tested this yet.
> This is a 1-in-the-morning patch attempt based on a 5-minute perusal of the
> source code. Please test before using!)", so rather than just slap it in I
> was hoping we could have a subject matter expert check it over...

... understood. Just in case, there's a koji scratch-build at [1], built from the source in [2] that includes the patch at comment #24.
 
> I'm not sure whether it's upstream appropriate, that's another thing someone
> more experienced in the area could say.

maybe we can share what we have as a reply to the thread [3] in the official hostap ML?
thanks!

-- 
davide

[1] https://koji.fedoraproject.org/koji/taskinfo?taskID=86457235
[2] https://src.fedoraproject.org/fork/dcaratti/rpms/wpa_supplicant/tree/bz2072070
[3] http://lists.infradead.org/pipermail/hostap/2022-April/040353.html

Comment 36 aland 2022-05-01 14:57:14 UTC
Just for clarification, can we name and shame the vendors who don't support secure renegotiation?

https://www.openssl.org/docs/man1.0.2/man3/SSL_get_secure_renegotiation_support.html says:

"OpenSSL 0.9.8m and later always attempts to use secure renegotiation as described in RFC5746."

There is literally nothing that the RADIUS server has to do in order to support this.  OpenSSl 0.9.8m was released in 2010.  There's no excuse for vendors to continue shipping such old software.

Also a minor nit:  It's not the " Wi-Fi vendor’s TLS server implementation", or the "EAP server".  It's the RADIUS server.  Many vendors have embedded RADIUS servers.  But I'm not aware of any EAP server which is *just* a RADIUS server.  All EAP servers are implemented 1-1 with RADIUS servers.

Comment 37 David Koppelman 2022-05-02 12:59:26 UTC
I tried the build of wpa_supplicant from Comment 35 (using the link below), and I'm able to connect. I had been experiencing the insecure handshake issue, and had applied several diagnostic configuration changes, some of which worked. I removed all of those diagnostic configuration changes and installed the new wpa_supplicant, and I can now connect.


[1] https://koji.fedoraproject.org/koji/taskinfo?taskID=86457235

Comment 38 Adam Williamson 2022-05-02 15:31:32 UTC
Thanks, David. As we're getting short on time again, if nobody who's an expert in the area chimes in soon, I'll review the patch myself as best I can and do a build today.

Comment 39 Fedora Update System 2022-05-02 18:19:18 UTC
FEDORA-2022-0c9603c2da has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-0c9603c2da

Comment 40 Angelo Theodorakis 2022-05-03 12:02:51 UTC
*** Bug 2076754 has been marked as a duplicate of this bug. ***

Comment 41 Fedora Update System 2022-05-03 13:32:08 UTC
FEDORA-2022-0c9603c2da has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-0c9603c2da`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-0c9603c2da

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 42 James Ralston 2022-05-04 07:28:19 UTC
I’m seeing this problem with RHEL9 beta, not Fedora 36 beta, so I can’t easily test the Fedora wpa_supplicant update, but for RHEL9 beta I rebuilt wpa_supplicant using the srpm package with the patch above attached, and now I can connect to our organization’s Wi-Fi network with RHEL9 beta, when before it failed. So I’m fairly confident the patch restores the ability to connect to Wi-Fi networks that use bad RADIUS server implementations that don’t implement RFC5746 secure renegotiation, which is the goal.

(In reply to Davide Caratti from comment #35)
> maybe we can share what we have as a reply to the thread [3] in the official
> hostap ML?

I did, here:

http://lists.infradead.org/pipermail/hostap/2022-May/040511.html

I will wait a few days, then poke upstream again.

(In reply to aland from comment #36)
> Just for clarification, can we name and shame the vendors who don't support
> secure renegotiation?

Our organization uses Arubet Networks for our enterprise Wi-Fi deployment, so other Aruba sites will likely hit this.

(In reply to aland from comment #36)
> Also a minor nit:  It's not the " Wi-Fi vendor’s TLS server implementation",
> or the "EAP server".  It's the RADIUS server.  Many vendors have embedded
> RADIUS servers.  But I'm not aware of any EAP server which is *just* a
> RADIUS server.  All EAP servers are implemented 1-1 with RADIUS servers.

Yes, thanks for the correction; I was not using good terminology.

Comment 43 Reese Armstrong (reesericci) 2022-05-04 11:19:07 UTC
Can confirm about Aruba. My school uses Aruba and that’s  where I saw the issue first crop up.

—reese

Comment 44 Neil Uranic 2022-05-04 13:51:17 UTC
I just tested this using a Fedora 36 Beta Laptop.  On initial boot the unit failed to connect to the wireless network.  Attempted several times, essentially a silent failure.  Updated wpa-supplicant to the patched version, rebooted and successfully connected.  I am on Arubet Networks.

Comment 45 Adam Williamson 2022-05-05 17:17:02 UTC
Thanks, Neil.

Comment 46 Fedora Update System 2022-05-05 18:37:18 UTC
FEDORA-2022-0c9603c2da has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 47 Michel Alexandre Salim 2022-05-05 19:09:42 UTC
Confirmed that this works fine on our office network (using Aruba)

Comment 48 James Ralston 2022-05-07 21:25:41 UTC
wpa_supplicant has now fixed this upstream, by adding an option to request SSL_OP_LEGACY_SERVER_CONNECT: 

https://lists.infradead.org/pipermail/hostap/2022-May/040522.html
https://w1.fi/cgit/hostap/commit/?id=566ce69a8d0e64093309cbde80235aa522fbf84e
https://w1.fi/cgit/hostap/commit/?id=a561d12d24c2c8bb0f825d4a3a55a5e47e845853

I think the option is global, so it can be used either for all networks:

phase1="allow_unsafe_renegotiation=1"

Or for a specific network:

network={
        ssid="dammit-Aruba"
        phase1="allow_unsafe_renegotiation=1"
}

Additionally, with these patches, wpa_supplicant now detects when PEAP fails because OpenSSL throws SSL_R_UNSAFE_LEGACY_RENEGOTIATION_DISABLED, and propagates that error. So it is now possible for NetworkManager to not only have a configuration option to enable phase1="allow_unsafe_renegotiation=1" on a per-interface basis, but to detect when a wireless network needs phase1="allow_unsafe_renegotiation=1" but it wasn’t supplied (e.g., because NetworkManager defaults to leaving it off, and the user didn’t check the appropriate checkbox in nm-connection-editor.

For now, probably the best thing to do is to ship Fedora 36 with the patched wpa_supplicant that unconditionally sets SSL_OP_LEGACY_SERVER_CONNECT. Once NetworkManager is updated to implement allowing the user to select phase1="allow_unsafe_renegotiation=1" on a per-network basis, then the patch can be dropped, and we will be where we want to be: if a user attempts to connect to a WiFi network whose RADIUS server doesn’t support RFC5746 secure renegotiation, they will be informed exactly why the connection failed, and how they can work around it. And as the operators of the WiFi networks get called out for failing to support RFC5746 secure renegotiation, they are going to turn around and complain to their vendors… and thus vendors that support neither TLSv1.3 or RFC5746 secure renegotiation are going to be shamed.

(Renegotiation was one of the many protocol features that TLSv1.3 deliberately dropped because it was more of a headache / security deficiency than it was useful. So if both wpa_supplicant and the RADIUS server support TLSv1.3, RFC5746 secure renegotiation is irrelevant.)

Maybe this will be ready in time for Fedora 37?

Comment 49 James Ralston 2022-05-07 21:42:34 UTC
BTW, I was remiss in pointing out: I approached wpa_supplicant upstream over this issue, and Jouni Malinen was fantastic in addressing it. I didn’t expect any movement on this for potentially months; he added the phase1="allow_unsafe_renegotiation=1" feature 6 days after I explained the issue on the hostap mailing list. Boom!

Thomas Haller, I have not delved into the NetworkManager source code, so alas, I am unable to assist with a patch. But since my organization uses Aruba, and I experience the problem, I will happily assist in testing once this feature lands in NetworkManager upstream.

Comment 50 Thomas Haller 2022-05-09 14:59:08 UTC
https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/1218 would make this opt-in in NetworkManager.

To detect the condition and do something about it is TODO (and much more work).
I think the only thing that NetworkManager should do when it detects the issue, is to make it clear to the user. So failing to do that, is bad UX/discoverability (and serious), but there is no other functional change. The problem for NetworkManager is that it has no good mechanism for sending informational messages to the NM clients. That needs improvements, but is a larger effort.

Comment 51 Stefan 2022-05-19 07:49:08 UTC
Good this this issue is being cared about. But that's only working around vendors that don't keep up with crypto, and the real fix should be to chase them. Knowing some Aruba people, I'd like to request some intel from the commenters above: in which releases did you encounter unsafe renegotiations being needed by the server side? Is this really happening on current releases or only old ones nearing EOL?


Note You need to log in before you can comment on or make changes to this bug.