Bug 2077973

Your RADIUS server does not send a renegotiation_info extension in its ServerHello message. See https://datatracker.ietf.org/doc/html/rfc5746. See specifically section 4.1, which discusses client behavior. OpenSSL 3 defaults to the secure client behavior, which requires the server to support RFC5746.

@dcaratti We should probably document this in the release notes.

See (the RHEL/Fedora equivalent of) https://bugs.launchpad.net/ubuntu/+source/wpa/+bug/1958267/comments/22, which should work around this issue.

Also note that this is entirely unrelated to the OpenSSL legacy provider. See https://bugzilla.redhat.com/show_bug.cgi?id=2069239#c24, which explains the difference between the OpenSSL legacy provider and the LEGACY crypto-policy. Additionally, "unsafe legacy renegotiation" is yet another unrelated use of the word "legacy" that has no relation to neither the OpenSSL provider, nor the LEGACY crypto-policy.

(In reply to Clemens Lang from comment #3)
> Your RADIUS server does not send a renegotiation_info extension in its
> ServerHello message. See https://datatracker.ietf.org/doc/html/rfc5746. See
> specifically section 4.1, which discusses client behavior. OpenSSL 3
> defaults to the secure client behavior, which requires the server to support
> RFC5746.
> 
> @dcaratti We should probably document this in the release notes.

I will test the work-around of setting the UnsafeLegacyRenegotiation openssl.cnf option, via a separate openssl.cnf file just for wpa_supplicant’s use.

I filed a ticket with our network team. They investigated the RADIUS server settings, to see if there was a setting to enable RFC5746 secure renegotiation that they had overlooked. Despite the fact that we are running a modern Wi-Fi infrastructure, using modern equipment, they could find no such setting. They filed an upstream bug report with the wireless vendor, but this now boils down to a feature request to our upstream vendor. In all likelihood, this will literally take years.

So, to recap:

1. The default RHEL9 beta OpenSSL configuration disables unsafe renegotiation, for all library contexts.

2. When a user attempts to connect to a Wi-Fi network that uses PEAP authentication and does not support secure renegotiation, NetworkManager behaves as if the user’s password was incorrect: no error message whatsoever is displayed to the user; NetworkManager simply prompts for the password again. So not only is there no indication what the actual problem was, the user is misled into thinking the issue is a mistyped password.

3. Even if the user is experienced enough to search the system logs, and discovers the “unsafe legacy renegotiation disabled” error message, there is literally *ZERO* information in the OpenSSL documentation as to what this error message means, let alone any information on how to actually enable unsafe legacy renegotiation.

And I do mean *ZERO*:

$ rpm -ql $(rpm -qa | grep openssl | sort | grep -v debug) | sort | uniq | ( while read F; do [ -f "${F}" ] && echo "${F}"; done ) | xargs -e -r zgrep -i unsafe
/usr/bin/grep: (standard input): binary file matches
/usr/bin/grep: (standard input): binary file matches
/usr/bin/grep: (standard input): binary file matches

The three binary matches are from the libssl.so.* shared libraries and the /usr/lib/.buildid/* file. None of the man pages nor any other documentation *even contains the word* “unsafe”.

4. Even if a user finally discovers via external information (e.g. web searches) how to work around this issue, the work-around involves manually modifying the systemd configuration for the wpa_supplicant service to point to a separate openssl.cnf file, which must also be manually created. From a UX perspective, this is unacceptable.

Yes, we want to stamp out bad security practices. But 1) the real-world security threat of insecure renegotiation being enabled in Wi-Fi PEAP authentication is inconsequential relative to the other security threats that users face, and 2) users have no control over whether the PEAP server supports RFC5746 secure renegotiation. If the long-term goal is to identify Wi-Fi networks where the PEAP servers fail to support RFC5746 secure renegotiation, then we need to enlist users to help achieve that goal, because users are the ones who are going to encounter such Wi-Fi networks. The UX needs to be crafted to achieve that goal.

As such, long-term, this issue should be solved as follows:

1. In the NetworkManager configuration, in the “Wi-Fi Security” tab, there is a “permit unsafe legacy TLS renegotiation in PEAP authentication” option. This option defaults to disabled; that is, PEAP authentication requires the TLS server to support RFC5746 secure renegotiation.

2. When invoking wpa_supplicant, NetworkManager passes the value of the “permit unsafe legacy TLS renegotiation in PEAP authentication” option.

3. Before wpa_supplicant performs PEAP authentication, if the “permit unsafe legacy TLS renegotiation in PEAP authentication” option was enabled, wpa_supplicant sets the SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION option on the OpenSSL ctx.

4. If wpa_supplicant did not set SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION, and PEAP authentication fails because OpenSSL returned error 0A000152 (unsafe legacy renegotiation disabled), wpa_supplicant communicates that specific error back to NetworkManager.

5. NetworkManager, upon receiving the “unsafe legacy renegotiation disabled but PEAP server does not support RFC5746 secure renegotiation” error from wpa_supplicant, throws a dialog box to inform the user that they must enable the “permit unsafe legacy TLS renegotiation in PEAP authentication” option for this Wi-Fi network, and that they might wish to consider notifying the owner of the Wi-Fi network that their Wi-Fi settings are insecure.

Yes, this is more work. But doing it this way changes the UX from “Red Hat fundamentally broke Wi-Fi in a myopic pursuit of security purity” to “hey, Red Hat let me know that this Wi-Fi network has a security issue than can be improved, and gave me an easy way to work around it”. For the former, Red Hat is the villain and will (deservedly) receive the blame; for the latter, Red Hat is the good guy and the Wi-Fi operator will be notified that they have a security issue to address.

But in the meantime, at the minimum, RHEL9 *must* ship with the “Options = UnsafeLegacyRenegotiation” openssl.cnf workaround enabled for wpa_supplicant.

If RHEL9 final ships with with wpa_supplicant requiring PEAP servers to support RFC5746 secure renegotiation, it is going to break widely and badly, in a way that not one user in a hundred will be able to even troubleshoot, let alone resolve. And because the vast majority of Wi-Fi clients (including previous versions of Fedora and RHEL) will work just fine, users will assume that the problem is that Red Hat shipped RHEL9 with broken Wi-Fi settings. (And they will be *absolutely correct* in that assumption.)

A few comments from <https://bugzilla.redhat.com/show_bug.cgi?id=2072070#c24>:

First: the UnsafeLegacyRenegotiation option (corresponding to SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION) is the wrong option. That will enable unsafe legacy renegotiation in *both* client and server contexts. That’s bad. The correct option to use here is UnsafeLegacyServerConnect (corresponding to SSL_OP_LEGACY_SERVER_CONNECT), which allows only TLS clients to connect to TLS servers that permit unsafe legacy renegotiation, while still causing server instances to reject a TLS client that does not support RFC5746 secure renegotiation.

Second, I strongly suspect this one-line patch will have the same effect as setting “Options = UnsafeLegacyServerConnect” in openssl.cnf:

diff -up wpa_supplicant-2.10/src/crypto/tls_openssl.c.legacy-server-connect wpa_supplicant-2.10/src/crypto/tls_openssl.c
--- wpa_supplicant-2.10/src/crypto/tls_openssl.c.legacy-server-connect  2022-01-16 15:51:29.000000000 -0500
+++ wpa_supplicant-2.10/src/crypto/tls_openssl.c        2022-04-28 02:47:26.863529683 -0400
@@ -1049,6 +1049,16 @@ void * tls_init(const struct tls_config
        SSL_CTX_set_options(ssl, SSL_OP_NO_SSLv2);
        SSL_CTX_set_options(ssl, SSL_OP_NO_SSLv3);
 
+       /* Many enterprise PEAP server implementations (e.g. used in large
+          corporations and universities) do not support RFC5746 secure
+          renegotiation, and starting with OpenSSL 3.0,
+          SSL_OP_LEGACY_SERVER_CONNECT is no longer set as part of SSL_OP_ALL.
+          So until we implement a way to request SSL_OP_LEGACY_SERVER_CONNECT
+          only in EAP peer mode, just set SSL_OP_LEGACY_SERVER_CONNECT
+          globally. */
+
+       SSL_CTX_set_options(ssl, SSL_OP_LEGACY_SERVER_CONNECT);
+
        SSL_CTX_set_mode(ssl, SSL_MODE_AUTO_RETRY);
 
 #ifdef SSL_MODE_NO_AUTO_CHAIN

(WARNING: I have not tested this yet. I will attempt to do so soon, and report back.)

Long term, as I wrote above, the best way to address this is to have NetworkManager and wpa_supplicant coordinate to detect when SSL_OP_LEGACY_SERVER_CONNECT is needed but not set, and advise the user to enable the (new) “permit unsafe legacy TLS renegotiation in PEAP authentication” option in the “Wi-Fi Security” tab for the Wi-Fi network in question.

But a patch something like the above (assuming it works) will squash the long-standing wpa_supplicant bug (failing to explicitly set SSL_OP_LEGACY_SERVER_CONNECT) that the OpenSSL 3.0 change of removing SSL_OP_LEGACY_SERVER_CONNECT from SSL_OP_ALL exposed.

Just FYI, I built wpa_supplicant from the wpa_supplicant-2.10-2.el9.src.rpm package with the patch in comment #7 applied, and now I can connect to Wi-Fi networks that previously failed.

(In reply to James Ralston from comment #9)
> Just FYI, I built wpa_supplicant from the wpa_supplicant-2.10-2.el9.src.rpm
> package with the patch in comment #7 applied, and now I can connect to Wi-Fi
> networks that previously failed.

hi James,

I'm about to fix this in RHEL-9.1 by backporting the following 2 commits:

https://w1.fi/cgit/hostap/commit/?id=a561d12d24c2c8bb0f825d4a3a55a5e47e845853
https://w1.fi/cgit/hostap/commit/?id=566ce69a8d0e64093309cbde80235aa522fbf84e

that will fix the reported issue for users that have text-file configuration. To get a complete fix, we will need to have the code in [2] merged in NetworkManager, that's going to be handled with a separate bugzilla.

Please note: given the number of users that might have hit this problem (that may become non negligible according to [3], until administrators of Aruba AP deploy a fix in the AP/Radius), this bug can be promoted to be fixed also in RHEL-9.0.z. I will file a z-stream request once the fix gets in 9.1.

thanks!
-- 
davide

[2] https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/1218
[3] https://bugzilla.redhat.com/show_bug.cgi?id=2072070#c42

Thanks Davide, all of that sounds reasonable.

I suspect Aruba will not be the only vendor whose non-HTTPS TLS shenanigans will finally be exposed by OpenSSL 3.0’s new defaults. HTTPS receives scrutiny (in no small part due to Qualsys SSL Labs), but many other protocols’ TLS implementations simply don’t receive the same degree of testing exposure. We plan to move aggressively to RHEL9 now that it is GA, so we will be vigilant in watching for further broken TLS implementations that OpenSSL 3.0 exposes.

(In reply to Davide Caratti from comment #10)
...
> configuration. To get a complete fix, we will need to have the code in [2]
> merged in NetworkManager, that's going to be handled with a separate
> bugzilla.
> 
> Please note: given the number of users that might have hit this problem
> (that may become non negligible according to [3], until administrators of
> Aruba AP deploy a fix in the AP/Radius), this bug can be promoted to be
> fixed also in RHEL-9.0.z. I will file a z-stream request once the fix gets
> in 9.1.

Do we have a 9.0.z request for the NM part already? I couldn't find it.

Just FYI, today our network team informed me:

> I just confirmed with Aruba that RFC5746 has been added to the recently released code. I’ll get this code running in our lab to see what all is required to enable it and then we’ll schedule when we can push this out to production.

So while it looks like upstream Wi-Fi vendors are finally beginning to feel the heat and take action, it will likely take years until a majority of sites are updated…

(In reply to Marcelo Ricardo Leitner from comment #17)
> (In reply to Davide Caratti from comment #10)
> ...
> > configuration. To get a complete fix, we will need to have the code in [2]
> > merged in NetworkManager, that's going to be handled with a separate
> > bugzilla.
> > 
> > Please note: given the number of users that might have hit this problem
> > (that may become non negligible according to [3], until administrators of
> > Aruba AP deploy a fix in the AP/Radius), this bug can be promoted to be
> > fixed also in RHEL-9.0.z. I will file a z-stream request once the fix gets
> > in 9.1.
> 
> Do we have a 9.0.z request for the NM part already? I couldn't find it.

I don't know if we have it; however, for 9.0.z there is also the possibility of doing a rhel-only patch, similar to the one we did for f36 [1], that does not require any change in the configuration. I think that the best thing is to agree on what to do with NetworkManager team, and then do the z-stream fix in wpa_supplicant in accordance with that decision.

[1] https://src.fedoraproject.org/rpms/wpa_supplicant/c/2a2d184805ae0712b13249fc9a7b58524a358164?branch=rawhide

I think the new wpa_supplicant option is the right solution to this bug, but I think it is misnamed. I will soon post some analysis in bug #2072070 where most of the discussion lives. Probably best to not create a 9.0.z update in the meantime.

hello Micheal, thanks for looking at this!
(In reply to Michael Catanzaro from comment #21)
> I think the new wpa_supplicant option is the right solution to this bug, but
> I think it is misnamed. I will soon post some analysis in bug #2072070 where
> most of the discussion lives. Probably best to not create a 9.0.z update in
> the meantime.

there is at least an issue with renaming a configuration: According to my understanding of [1], we can't do that in the middle of a RHEL X stream. So, unless we rename the parameter before the 9.1 code-freeze,we will have to live with the "misnamed" parameter. 

(In reply to Davide Caratti from comment #20)
> (In reply to Marcelo Ricardo Leitner from comment #17)
> > 
> > Do we have a 9.0.z request for the NM part already? I couldn't find it.
> 
> I don't know if we have it; however, for 9.0.z there is also the possibility
> of doing a rhel-only patch, similar to the one we did for f36 [1], that does
> not require any change in the configuration. I think that the best thing is
> to agree on what to do with NetworkManager team, and then do the z-stream
> fix in wpa_supplicant in accordance with that decision.

Got a feedback from NM team: users will surely complain if WiFi connectivity with EAP-TTLS starts having troubles with 9.1, while everything is ok in 9.0.z and 8.y. The above proposal (or a slightly better variant of it (*)) is the best thing to do for preparing RHEL-8 to RHEL-9 migration of existing profiles.

(*) a good variant of the above proposals is to keep the configuration parameter, misnamed or not, but change wpa_supplicant with a small (RHEL-only) patch that sets TLS_CONN_ALLOW_UNSAFE_RENEGOTIATION bit when the "allow_unsafe_renegotiation" parameter  is not specified in the configuration. Newly-created EAP-TTLS connections will explicitly have this parameter set to 0 (by default, or set to 1 in case of user intervention), while older profiles will keep working seamlessly, even with unsafe RADIUS servers, in the transition between rhel-8 and rhel-9. Any opinion?

https://w1.fi/cgit/hostap/commit/?id=566ce69a8d0e64093309cbde80235aa522fbf84e

[1] https://one.redhat.com/rhel-development-guide/#con_rhel-stability_assembly_development

(In reply to Davide Caratti from comment #22)
> there is at least an issue with renaming a configuration: According to my
> understanding of [1], we can't do that in the middle of a RHEL X stream. So,
> unless we rename the parameter before the 9.1 code-freeze,we will have to
> live with the "misnamed" parameter. 

Definitely. Accordingly, I would encourage you to fix the name of this option before we hit the code freeze, so you don't get locked into supporting it for the next decade and more.

Or alternatively, you could change the option to actually do what it says and toggle SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION instead of SSL_OP_LEGACY_SERVER_CONNECT. But, as I've explained in bug #2072070, that seems like an inferior solution to this bug.

NetworkManager should detect whenever wpa_supplicant calls eap_notify_status(sm, "unsafe server renegotiation", "failure") and then do one of the following:

1. Automatically retry wpa_supplicant with allow_unsafe_renegotiation=1, without any user notification.

2. Advise the user that the connection attempt will succeed if the user sets the (e.g.) “permit unsafe legacy server connections (no RFC5746)” NetworkManager option for the connection.

The motive for #2 is to ensure that the user is OK with connecting to a Wi-Fi network with known security deficiencies, but an ulterior motive is to motivate users to pressure the operators of the Wi-Fi network to correct those deficiencies.

In terms of the name of the option, I asked hostap upstream if they are willing to rename it:

http://lists.infradead.org/pipermail/hostap/2022-July/040665.html

If they say yes, I will submit a patch to upstream to rename the option name from “allow_unsafe_renegotiation” to “allow_legacy_server_connect”.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (wpa_supplicant bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8287