Bug 1630260 - Removal of component glib2 or its crypto from RHEL
Summary: Removal of component glib2 or its crypto from RHEL
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: glib2
Version: 8.1
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 8.1
Assignee: Colin Walters
QA Contact: Desktop QE
URL:
Whiteboard:
Depends On: 1680703 1718074
Blocks: 1635116 1896162
TreeView+ depends on / blocked
 
Reported: 2018-09-18 09:41 UTC by Nikos Mavrogiannopoulos
Modified: 2021-01-06 17:53 UTC (History)
12 users (show)

Fixed In Version: glib2-2.56.4-4.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1896162 (view as bug list)
Environment:
Last Closed: 2019-11-05 22:05:45 UTC
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1637801 unspecified CLOSED pidgin: do not reimplement HMAC-SHA1 2020-11-14 13:31:20 UTC
Red Hat Product Errata RHSA-2019:3530 None None None 2019-11-05 22:05:59 UTC

Internal Links: 1637801

Description Nikos Mavrogiannopoulos 2018-09-18 09:41:38 UTC
Description of problem:
This component brings new implementations of cryptographic algorithms in RHEL, which affects the manageability and consistency of the operating system, and voids our message on the operating system’s FIPS140-2 certification and compliance. This bug is about removing this component from RHEL, and any dependent applications should be modified to depend on RHEL core crypto libraries (see https://access.redhat.com/articles/3359851 )

In particular this component implements HMAC:
https://pelc.engineering.redhat.com/reviews/6016/#crypto

Comment 1 Colin Walters 2018-09-18 15:28:24 UTC
A whole lot of things link to libglib - if we start depending on one of openssl or gnutls it'll force a lot of other things to link to them too.

Maybe we could just dlopen() openssl as soon as the HMAC API is used.

Whatever we do here, if we don't do it upstream it's going to be a substantial maintenance burden.

Comment 2 Nikos Mavrogiannopoulos 2018-09-21 07:12:05 UTC
Is that HMAC used for a purpose that can be interpreted as protecting user data (through authentication for example)? If its use is not security-related, maybe it makes sense to give this package an exception rather than convoluting the code.

Comment 3 Colin Walters 2018-09-21 12:23:03 UTC
Nothing in GLib itself uses the GHmac API.  However, a quick perusal of: https://codesearch.debian.net/search?q=g_hmac_new&perpkg=1 shows its usage in a bunch of applications, from qemu to shotwell to cockpit.

Comment 4 Nikos Mavrogiannopoulos 2018-10-02 07:26:48 UTC
Ok. The options for RHEL8 are:
 * Mark glib as non-compliant for crypto
 * Modify glib to use a crypto library from the set (openssl, gnutls, nss)

I think the first option will raise eyebrows (why wouldn't glib be recommended for crypto?)

Comment 6 Nikos Mavrogiannopoulos 2018-10-02 07:28:55 UTC
> A whole lot of things link to libglib - if we start depending on one of openssl or gnutls it'll force a lot of other things to link to them too.

> Maybe we could just dlopen() openssl as soon as the HMAC API is used.

Is the linking problem you mention something we should worry in RHEL? If not why not make that part of the code conditional and only enable it on rhel?

Comment 9 Simo Sorce 2019-03-27 15:11:20 UTC
Hi,
this bug is marked as blocking #1635116 which means the component is implementing cryptogrpahic algorithms it shouldn't, and that is a problem for our RHEL security certifications.

As of recently the FIPS compliance project has been transferred to me (Simo), so I would like to asses the status of all the components that still have bugs open.

As the RHEL 8.1 deadlines are looming it is important that we understand what's the status of these bugs, resolution ETA or additional blockers that need resolving.

This update request comes in now as I would like to avoid last minute escalations or exception requests.

Thank you for your cooperation.
Simo.

Comment 10 Paolo Bonzini 2019-04-02 08:33:28 UTC
For what it's worth, QEMU does not use g_hmac_new in RHEL8 (it uses libgcrypt's HMAC implementation).

Comment 11 Colin Walters 2019-04-02 14:29:27 UTC

> Is the linking problem you mention something we should worry in RHEL? If not why not make that part of the code conditional and only enable it on rhel?

Changing glib2 to link to e.g. openssl would be a quite major thing to carry just in RHEL.  People would find it surprising.

We have to think about cases where e.g. an application is using openssl-deprecated and glib2 - I wouldn't be at all surprised if something like this exists.

Can we just audit for any uses of g_hmac_*() in RHEL and verify there aren't any?

Comment 12 Simo Sorce 2019-04-02 14:36:21 UTC
I think we have a few ooptions here:
(1) make glib2 link to an approved cryptographic library on RHEL (I do not think this would be surprising anymore than any other library doing it).
(2) disable the code in glib2 so that if something tries to use it, it will fail.
(3) remove the API completely

Of the three options it looks to me (1) is the least disruptive.
However from my POV any of these (or any other proposal) is fine as long as in the end glib2 is not implementing cryptographic functions itself.

Comment 13 Colin Walters 2019-04-02 17:13:31 UTC
> (2) disable the code in glib2 so that if something tries to use it, it will fail.

Can we do this only in FIPS mode or something?

Comment 14 Simo Sorce 2019-04-02 18:01:18 UTC
HMAC is an approved API in FIPS, so there isn't a strict need to disable it, also you would need to implement code to detect if the system is in FIPS mode and we would then have tests that insures it effectively works, I would rather not do that it is quite some work.

What is the issue with linking to an existing RHEL supported cryptographic library?
Is the problem that you think upstream would be against it and would have to make and maintain a downstream patch?

Comment 15 Colin Walters 2019-04-04 14:22:57 UTC
> What is the issue with linking to an existing RHEL supported cryptographic library?

I wrote this above:

We have to think about cases where e.g. an application is using compat-openssl and glib2 - I wouldn't be at all surprised if something like this exists.

This will happen for e.g. an app built against RHEL7 and the user is running it in RHEL8.  That works fine today - I have actually done that with ostree which is today linked against both openssl and glib2.
We can't have both OpenSSL in the same process right?

I guess we could pick gnutls or nss to avoid this, but then we're forcing a big swath of apps to link against an extra crypto library.

> Is the problem that you think upstream would be against it and would have to make and maintain a downstream patch?

I think we could probably get this upstream as a build time option, but no matter what we do since glib2 is so widely used it will have ecosystem impact.  Think about people who write binaries they want to have work across Ubuntu and Fedora/RHEL for example.

Comment 16 Simo Sorce 2019-04-04 15:01:14 UTC
Is glib2 a level1 library? (asking to figure out if runnign an app linked against glib2 in RHEL7 is actually supported in RHEL8)

Anyway, normally you cannot link 2 copies of two version of the same library in the same process unless you use symbol versioning.
But it is possible we may do that via dlopen() and RTLD_LOCAL and RTLD_DEEPBIND.
I realize having to use dlopen() instead of just linking at build time is not the best, and it would be nicer if we could link to openssl without causing this to bleed in the application. I'll ask glibc developers if that is possible.

Comment 17 Simo Sorce 2019-04-04 15:58:18 UTC
So although there is theorethically the -Bgroup option to isolate a library dependencies apparently it is not fully implemented yet so we can't use it.

So thinking more about RHEL and our support statement, if glib is not a level1 library we do not make any promises that something built on RHEL7 will work with the glib built on RHEL8, so I wonder if we should really care about that case.

However if we should (or even just want to) care, then using dlopen() is a possibility. Is that an option you would be ok pursuing?

Comment 18 Colin Walters 2019-04-04 16:18:51 UTC
It looks like the ACG guide is...broken, according to https://access.redhat.com/articles/rhel-abi-compatibility#Appendix

glib2 is ACG2 *but* gtk2 is ACG1, and gtk2 depends on a whole lot of stuff, including glib2; there is no feasible way we could support gtk2 ACG1 but not glib2.

glib2 is widely used in the ecosystem - on the server side, we ship pacemaker, qemu, ostree/rpm-ostree.  On the desktop side, obviously a ton.
And I know there are just a ton of 3rd party apps.

Comment 19 Colin Walters 2019-04-05 13:21:49 UTC
FWIW this code came from https://gitlab.gnome.org/GNOME/glib/commit/acbcb8f7e37b1c0ff872fdbb74df7de690bc2c6f

Comment 20 Stef Walter 2019-04-08 10:58:07 UTC
Hmmm, isn't the Glib code in glib/gchecksum.c (MD5, SHA1, SHA256, SHA512) also crypto?

In p11-kit I remember we linked to freebl3 to check the "doesn't implement own crypto" checkbox. It's awkward slow and hacky, but it was necessary to placate RHEL requirements. Does a similar solution work here?

https://github.com/p11-glue/p11-kit/blob/master/trust/digest.c#L44

Comment 21 Nikos Mavrogiannopoulos 2019-04-08 14:16:33 UTC
(In reply to Stef Walter from comment #20)
> Hmmm, isn't the Glib code in glib/gchecksum.c (MD5, SHA1, SHA256, SHA512)
> also crypto?
> 
> In p11-kit I remember we linked to freebl3 to check the "doesn't implement
> own crypto" checkbox. It's awkward slow and hacky, but it was necessary to
> placate RHEL requirements. Does a similar solution work here?
> 
> https://github.com/p11-glue/p11-kit/blob/master/trust/digest.c#L44

Hashes are no longer considered crypto, and there is no need to have p11-kit with freebl (thankfully!). Hmac is a different story as it is a keyed mac.

Comment 23 Colin Walters 2019-06-06 14:56:50 UTC
A few options I see for this:

1) Add glib's HMAC implementation to the validated list

2) Write patch to have glib's HMAC redirect to nss/openssl

3) Verify that no packages shipped in RHEL use this HMAC API, and *disable* it in FIPS mode

Any other variants I'm missing?

I can quickly try to gather some data on 3) as that seems clearly simplest.  Though of course none of these are mutually exclusive, we can start with 3) and go to 1) or 2) later.

Comment 24 Colin Walters 2019-06-06 15:00:58 UTC
On 3) though at least:

```
walters@toolbox ~/s/g/c/cockpit> rg -i g_hmac
src/ssh/cockpitsshknownhosts.c
376:  hmac = g_hmac_new (G_CHECKSUM_SHA1, (guchar *)copied, salt_length);
377:  g_hmac_update (hmac, (guchar *)host, strlen (host));
378:  g_hmac_get_digest (hmac, generated_hash, &generated_length);
385:    g_hmac_unref (hmac);
```

Comment 25 Colin Walters 2019-06-06 15:04:26 UTC
I think I am going to work on 3.  It's the most conservative option, but per above it will break Cockpit (and potentially other apps) until they port to another crypto library.

I'd be interested in the crypto team's estimate for the cost of 1).

Comment 26 Colin Walters 2019-06-06 15:11:32 UTC
Does anyone object to making this bug public so the upstream glib developers (not all of whom are RHT employees) can see?

Comment 27 Colin Walters 2019-06-06 15:23:52 UTC
Currently `ghmac.c` weighs in at 444 lines of code, and just for fun I ran `sloccount` against it which says:

Total Physical Source Lines of Code (SLOC)                = 230
Development Effort Estimate, Person-Years (Person-Months) = 0.04 (0.51)

That's...not very much code it seems to me.  (But I can imagine that verification is a nontrivial cost, again, not very much code here)

Comment 28 Simo Sorce 2019-06-06 16:37:27 UTC
(In reply to Colin Walters from comment #23)
> A few options I see for this:
> 
> 1) Add glib's HMAC implementation to the validated list

Can you explain what does this mean?
We are not expanding the number of packages we are doing FIPS validation for, we are actually shrinking it by moving all crypto operations in the crypto libraries
FIPS validation is an extremely expensive process (both in terms of time and money), and would require you to do 20x the work you have now (you need to run self tests at runtime, and a bunch of other things).

> 2) Write patch to have glib's HMAC redirect to nss/openssl

Yes please.

> 3) Verify that no packages shipped in RHEL use this HMAC API, and *disable*
> it in FIPS mode

This is also fine.

> Any other variants I'm missing?
> 
> I can quickly try to gather some data on 3) as that seems clearly simplest. 
> Though of course none of these are mutually exclusive, we can start with 3)
> and go to 1) or 2) later.

If there are users you can also ask them to swith to openssl and still do 3, however this analysis is coming very late, devel freeze is next week and it would be their right to push back to you.

Comment 29 Colin Walters 2019-06-06 19:44:40 UTC
https://gitlab.gnome.org/GNOME/glib/merge_requests/897

$ cat ghmac.c
#include <glib.h>

int
main (int argc, char **argv)
{
    const char key[] = "42";
    const char val[] = "foo";
    GHmac *hmac = g_hmac_new (G_CHECKSUM_SHA1, key, sizeof (key));
    g_hmac_update (hmac, val, sizeof (val));
    g_print ("%s\n", g_hmac_get_string (hmac));
}
$ ./ghmac
322a4aa0856e64f2f1dc2b35b02ce249a9744b43
$ echo 1 > /tmp/one
$ sudo mount --bind /tmp/one /proc/sys/crypto/fips_enabled 
$ ./ghmac 
(process:695): GLib-ERROR **: 19:40:50.190: g_hmac_new() is not available in FIPS mode

Comment 30 Colin Walters 2019-06-06 19:50:48 UTC
I guess a variant of 2) is "detect FIPS mode and dlopen() openssl".  That would push back the problem of glib linking a crypto library to FIPS mode only.

I really have to emphasize the large scope of glib pulling in a crypto library.  A whole lot of stuff links to glib, from Firefox/Chrome, qemu, ostree, pacemaker, and last I looked VMWare Workstation, etc.
And my #1 concern here is openssl-deprecated vs openssl.
Today Chrome links BoringSSL statically because otherwise: https://bugzilla.redhat.com/show_bug.cgi?id=1448649

Comment 31 Martin Pitt 2019-06-06 20:41:03 UTC
For the record, cockpit does *not* use GHMac in RHEL 8 (only in RHEL 7). See https://bugzilla.redhat.com/show_bug.cgi?id=1718074#c2 for details.

Comment 32 Colin Walters 2019-06-06 20:45:44 UTC
I verified at least that our build of qemu is not using GHmac; qemu has an internal crypto abstraction and RHEL8's build uses gcrypt:
http://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/qemu-kvm/4.0.0/3.module+el8.1.0+3265+26c4ed71/data/logs/x86_64/build.log

Comment 33 Simo Sorce 2019-06-06 21:00:31 UTC
(In reply to Colin Walters from comment #30)
> I guess a variant of 2) is "detect FIPS mode and dlopen() openssl".  That
> would push back the problem of glib linking a crypto library to FIPS mode
> only.
> 
> I really have to emphasize the large scope of glib pulling in a crypto
> library.  A whole lot of stuff links to glib, from Firefox/Chrome, qemu,
> ostree, pacemaker, and last I looked VMWare Workstation, etc.
> And my #1 concern here is openssl-deprecated vs openssl.
> Today Chrome links BoringSSL statically because otherwise:
> https://bugzilla.redhat.com/show_bug.cgi?id=1448649

I know it is sometimes a problem to use a cryptolibrary from a library due to dependencies.
The best course of action is to not implement cyrptographic functions in glib.
Can't you simply deprecate upstream and compile out gmac in RHEL ?

Is any RHEL package actually using it?

If not, it would probably be better to simply patch it out in the RHEL packages so that people will see failures at build time instead of at runtime only in FIPS mode.

In general it is bad to have crypto implemented outside of crypto libraries even when not running in FIPS mode because these applications then are not following crypto-policy configurations.

For example we'll soon disable SHA-1 by default everywhere and glib would happily continue to allow it's use in non-FIPS mode, this is inconsistent with our promise to allow the dmin to control the crypto policies system wide and another reason why we ask package to use crypto-libraries where possible.

Comment 34 Colin Walters 2019-06-06 21:02:40 UTC
I mentioned above shotwell; that was imported into rhel-8 dist-git but looks like it was decided not to ship it.

Then, what we come down to here is the "not in RHEL" code, particularly proprietary code not turned up by the code search.  We can force *all* apps linking to glib to also link to a crypto library, or we can break just ones using GHmac.  Given the data so far I'm strongly leaning towards the latter.

So I'll be applying https://gitlab.gnome.org/GNOME/glib/merge_requests/897 to dist-git soon.  (Code review would be cool if anyone feels up for it...)

Comment 36 Colin Walters 2019-06-06 21:06:37 UTC
> If not, it would probably be better to simply patch it out in the RHEL packages so that people will see failures at build time instead of at runtime only in FIPS mode.

But that would break 3rd party apps using it even when not in FIPS mode.  I'm not so sure about doing that.  Again there's just so much glib-using code out there.  I didn't mention GStreamer before, think about people doing custom media streaming servers/devices; if they happen to use GHmac for something...

Maybe an incremental middle ground here (and one upstreamable likely) is to deprecate the API, so people at least see warnings.  Then we can revisit trying to disable it after people have hopefully been seeing compiler warnings for a while.

Comment 37 Simo Sorce 2019-06-06 21:14:03 UTC
I took a look at the change, it looks "ok" as far as hacks go, however it will probbaly cause bugs if there is any application that really use it, because peolple will rightfully complain that you are denying "approved" HMACs in FIPS mode, and that is unexpected.
 I am ok breking only apps that use GHmac, but I would rather have them break (also) at compile time, and not just at runtime.

Perhaps we can come to a compromise where we disable GHmac in the header files, so rebuilds will flag it out with a build failure (maybe with a #define around it that can re-enable the API if *really* someone needs it), but we keep the actual symbol in the library with your patch to disable it in FIPS mode ?

Comment 38 Stef Walter 2019-06-07 09:23:49 UTC
Colin, the documentation for g_hmac_new() allows returning %NULL, and does so for non-OOM conditions. For example if a digest is not supported. In FIPS mode, the entire scheme (HMAC) is not supported ... so %NULL seems appropriate no?

Comment 39 Colin Walters 2019-06-07 12:35:28 UTC
> Colin, the documentation for g_hmac_new() allows returning %NULL

As I commented in the patch, that's true but...at least Cockpit's code didn't check for NULL and so it would just result in a segfault rather than having a useful error message by default.

The Vala-generated C code that Shotwell is using also doesn't check for NULL.

Comment 41 Debarshi Ray 2019-06-07 13:24:31 UTC
(In reply to Simo Sorce from comment #33)
> Is any RHEL package actually using it?

In bug 1637801, pidgin is about to get switched to use GLib's GHmac instead of its own hand rolled copy. That's also what Pidgin development snapshots do.

I don't think it's reasonable to just yank out the entire GHmac API from glib2 and expect existing GLib users to figure it out on their own.

Comment 42 Simo Sorce 2019-06-07 17:13:34 UTC
Look you have a choice,
remove it, or link to an approved crypto library.

In the *very short* term, Colin's patch is somewhat acceptable, but it is not a long term fix, as it actually breaks any use of the library (and of depending applications) in FIPS mode, and does not conforms to crypto-policies in non-FIPS mode.

So it is a hack that is allowable only for 8.1 because we are short on time, it is not a solution past 8.1 and will need more work for 8.2 if you go with this one in 8.1 

If you can't remove the API, and there are actual RHEL users of the API, then the longer term solution is to link to a crypto library either directly or via dlopen() every time this API is used. I do not think it is in spirit to link only when in FIPS mode.

It is really unfortunate glib painted itself in a corner by implementing cryptographic functions, but we need to solve this issue properly, we can't live with hacks long term.

Comment 43 Debarshi Ray 2019-06-07 17:34:38 UTC
(In reply to Simo Sorce from comment #42)
> It is really unfortunate glib painted itself in a corner by implementing
> cryptographic functions, but we need to solve this issue properly, we can't
> live with hacks long term.

The fact remains that directly using any of those cryptographic libraries is a pain that lot of upstreams don't want to deal with. There are all kinds of things out there that were rolling their own HMAC implementation, and that's why GHmac was added to GLib.

It's not like people add random API to a library that can never ever break API/ABI ever for fun. :)

A cursory search brought up folks, pidgin, rest, shotwell, Telepathy (when it was still relevant). There's probably more. Some of these used to be a part of RHEL, some aren't today, but the fact is that there are real users of it out there.

See https://bugzilla.gnome.org/show_bug.cgi?id=652480 for the full discussion about why GHmac was added.

Anyway, I'd leave it to the GLib maintainers to figure out a way forward. I can imagine GLib adding an extension point that can be backed by GnuTls and OpenSSL via glib-networking and glib-openssl [1] respectively.

But saying that nobody can ever touch GHmac doesn't look like a realistic solution to me. We'd be stuck writing and maintaining downstream patches for random modules forever.

[1] https://gitlab.gnome.org/Archive/glib-openssl

Comment 44 Ray Strode [halfline] 2019-06-07 18:12:06 UTC
Hi,

(In reply to Colin Walters from comment #15)
> I guess we could pick gnutls or nss to avoid this, but then we're forcing a
> big swath of apps to link against an extra crypto library.
why is that a problem? They're probably already loaded in some running program anyway, and the mappings should be shared between processes, so it doesn't really seem like that big of a deal?

Comment 45 Simo Sorce 2019-06-07 18:20:00 UTC
Ok, so I am confused now.
Rishi and Ray tell me glib2 already links dynamically to cryptographic libraries, so why is it a problem to use also HMAc from either gnutls or openssl ?

Comment 46 Ray Strode [halfline] 2019-06-07 18:29:04 UTC
the problem is glib is more than one library. g_hmac is in libglib, and the gnutls/opessl stuff is from plugins loaded by libgio.  the abstraction over dlopen() is done by libgmodule which libgio links to but libglib doesn't.

of course, most programs that use glib link against all three, so libglib could use weak symbols or equivalent I guess...but that's kind of ugly. I wish we'd just declare gnutls a winner here and link against it directly without extension points and dlopen etc, maybe with a build flag to disable linking to it for distros that really can't have the dependency.

Comment 47 Colin Walters 2019-06-07 18:37:11 UTC
> Rishi and Ray tell me glib2 already links dynamically to cryptographic libraries, so why is it a problem to use also HMAc from either gnutls or openssl ?

That's only if glib-networking is loaded; it isn't in all cases.  That said my primary concern was about using openssl...it's true that it's going to be very common for glib apps to also pull in glib-networking which uses gnutls (today...there is a patch to port it to openssl which brings back a can of worms but we can ignore that for now).

I will spend some time today investigating a --enable-hmac-gnutls build-time option for glib which replaces the ghmac API with static inlines that redirect to gnutls.

Comment 58 errata-xmlrpc 2019-11-05 22:05:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:3530

Comment 61 Michael Catanzaro 2020-11-09 19:15:46 UTC
FYI: this was never upstreamed to Fedora, let alone glib upstream, so it won't be included in RHEL 9.

Comment 62 Simo Sorce 2020-11-09 20:36:57 UTC
Michael,
requirements haven't changed in RHEL-9,
do you need a clone of this bug ?

Comment 63 Michael Catanzaro 2020-11-09 20:42:10 UTC
A clone for RHEL 9 wouldn't hurt.

Looks like back before I joined Red Hat, I agreed this could be added to Fedora if the downstream patch was updated slightly (to use gnutls_hmac_copy() to implement g_hmac_copy(), rather than crashing if that function is called). So we could just do that.

Upstream glib is going to be harder, though.


Note You need to log in before you can comment on or make changes to this bug.