Bug 1630260

A whole lot of things link to libglib - if we start depending on one of openssl or gnutls it'll force a lot of other things to link to them too.

Maybe we could just dlopen() openssl as soon as the HMAC API is used.

Whatever we do here, if we don't do it upstream it's going to be a substantial maintenance burden.

Is that HMAC used for a purpose that can be interpreted as protecting user data (through authentication for example)? If its use is not security-related, maybe it makes sense to give this package an exception rather than convoluting the code.

Nothing in GLib itself uses the GHmac API.  However, a quick perusal of: https://codesearch.debian.net/search?q=g_hmac_new&perpkg=1 shows its usage in a bunch of applications, from qemu to shotwell to cockpit.

Ok. The options for RHEL8 are:
 * Mark glib as non-compliant for crypto
 * Modify glib to use a crypto library from the set (openssl, gnutls, nss)

I think the first option will raise eyebrows (why wouldn't glib be recommended for crypto?)

> A whole lot of things link to libglib - if we start depending on one of openssl or gnutls it'll force a lot of other things to link to them too.

> Maybe we could just dlopen() openssl as soon as the HMAC API is used.

Is the linking problem you mention something we should worry in RHEL? If not why not make that part of the code conditional and only enable it on rhel?

Hi,
this bug is marked as blocking #1635116 which means the component is implementing cryptogrpahic algorithms it shouldn't, and that is a problem for our RHEL security certifications.

As of recently the FIPS compliance project has been transferred to me (Simo), so I would like to asses the status of all the components that still have bugs open.

As the RHEL 8.1 deadlines are looming it is important that we understand what's the status of these bugs, resolution ETA or additional blockers that need resolving.

This update request comes in now as I would like to avoid last minute escalations or exception requests.

Thank you for your cooperation.
Simo.

For what it's worth, QEMU does not use g_hmac_new in RHEL8 (it uses libgcrypt's HMAC implementation).

> Is the linking problem you mention something we should worry in RHEL? If not why not make that part of the code conditional and only enable it on rhel?

Changing glib2 to link to e.g. openssl would be a quite major thing to carry just in RHEL.  People would find it surprising.

We have to think about cases where e.g. an application is using openssl-deprecated and glib2 - I wouldn't be at all surprised if something like this exists.

Can we just audit for any uses of g_hmac_*() in RHEL and verify there aren't any?

I think we have a few ooptions here:
(1) make glib2 link to an approved cryptographic library on RHEL (I do not think this would be surprising anymore than any other library doing it).
(2) disable the code in glib2 so that if something tries to use it, it will fail.
(3) remove the API completely

Of the three options it looks to me (1) is the least disruptive.
However from my POV any of these (or any other proposal) is fine as long as in the end glib2 is not implementing cryptographic functions itself.

> (2) disable the code in glib2 so that if something tries to use it, it will fail.

Can we do this only in FIPS mode or something?

HMAC is an approved API in FIPS, so there isn't a strict need to disable it, also you would need to implement code to detect if the system is in FIPS mode and we would then have tests that insures it effectively works, I would rather not do that it is quite some work.

What is the issue with linking to an existing RHEL supported cryptographic library?
Is the problem that you think upstream would be against it and would have to make and maintain a downstream patch?

> What is the issue with linking to an existing RHEL supported cryptographic library?

I wrote this above:

We have to think about cases where e.g. an application is using compat-openssl and glib2 - I wouldn't be at all surprised if something like this exists.

This will happen for e.g. an app built against RHEL7 and the user is running it in RHEL8.  That works fine today - I have actually done that with ostree which is today linked against both openssl and glib2.
We can't have both OpenSSL in the same process right?

I guess we could pick gnutls or nss to avoid this, but then we're forcing a big swath of apps to link against an extra crypto library.

> Is the problem that you think upstream would be against it and would have to make and maintain a downstream patch?

I think we could probably get this upstream as a build time option, but no matter what we do since glib2 is so widely used it will have ecosystem impact.  Think about people who write binaries they want to have work across Ubuntu and Fedora/RHEL for example.

Is glib2 a level1 library? (asking to figure out if runnign an app linked against glib2 in RHEL7 is actually supported in RHEL8)

Anyway, normally you cannot link 2 copies of two version of the same library in the same process unless you use symbol versioning.
But it is possible we may do that via dlopen() and RTLD_LOCAL and RTLD_DEEPBIND.
I realize having to use dlopen() instead of just linking at build time is not the best, and it would be nicer if we could link to openssl without causing this to bleed in the application. I'll ask glibc developers if that is possible.

So although there is theorethically the -Bgroup option to isolate a library dependencies apparently it is not fully implemented yet so we can't use it.

So thinking more about RHEL and our support statement, if glib is not a level1 library we do not make any promises that something built on RHEL7 will work with the glib built on RHEL8, so I wonder if we should really care about that case.

However if we should (or even just want to) care, then using dlopen() is a possibility. Is that an option you would be ok pursuing?

It looks like the ACG guide is...broken, according to https://access.redhat.com/articles/rhel-abi-compatibility#Appendix

glib2 is ACG2 *but* gtk2 is ACG1, and gtk2 depends on a whole lot of stuff, including glib2; there is no feasible way we could support gtk2 ACG1 but not glib2.

glib2 is widely used in the ecosystem - on the server side, we ship pacemaker, qemu, ostree/rpm-ostree.  On the desktop side, obviously a ton.
And I know there are just a ton of 3rd party apps.

FWIW this code came from https://gitlab.gnome.org/GNOME/glib/commit/acbcb8f7e37b1c0ff872fdbb74df7de690bc2c6f

Hmmm, isn't the Glib code in glib/gchecksum.c (MD5, SHA1, SHA256, SHA512) also crypto?

In p11-kit I remember we linked to freebl3 to check the "doesn't implement own crypto" checkbox. It's awkward slow and hacky, but it was necessary to placate RHEL requirements. Does a similar solution work here?

https://github.com/p11-glue/p11-kit/blob/master/trust/digest.c#L44

(In reply to Stef Walter from comment #20)
> Hmmm, isn't the Glib code in glib/gchecksum.c (MD5, SHA1, SHA256, SHA512)
> also crypto?
> 
> In p11-kit I remember we linked to freebl3 to check the "doesn't implement
> own crypto" checkbox. It's awkward slow and hacky, but it was necessary to
> placate RHEL requirements. Does a similar solution work here?
> 
> https://github.com/p11-glue/p11-kit/blob/master/trust/digest.c#L44

Hashes are no longer considered crypto, and there is no need to have p11-kit with freebl (thankfully!). Hmac is a different story as it is a keyed mac.

A few options I see for this:

1) Add glib's HMAC implementation to the validated list

2) Write patch to have glib's HMAC redirect to nss/openssl

3) Verify that no packages shipped in RHEL use this HMAC API, and *disable* it in FIPS mode

Any other variants I'm missing?

I can quickly try to gather some data on 3) as that seems clearly simplest.  Though of course none of these are mutually exclusive, we can start with 3) and go to 1) or 2) later.

On 3) though at least:

```
walters@toolbox ~/s/g/c/cockpit> rg -i g_hmac
src/ssh/cockpitsshknownhosts.c
376:  hmac = g_hmac_new (G_CHECKSUM_SHA1, (guchar *)copied, salt_length);
377:  g_hmac_update (hmac, (guchar *)host, strlen (host));
378:  g_hmac_get_digest (hmac, generated_hash, &generated_length);
385:    g_hmac_unref (hmac);
```

I think I am going to work on 3.  It's the most conservative option, but per above it will break Cockpit (and potentially other apps) until they port to another crypto library.

I'd be interested in the crypto team's estimate for the cost of 1).

Does anyone object to making this bug public so the upstream glib developers (not all of whom are RHT employees) can see?

Currently `ghmac.c` weighs in at 444 lines of code, and just for fun I ran `sloccount` against it which says:

Total Physical Source Lines of Code (SLOC)                = 230
Development Effort Estimate, Person-Years (Person-Months) = 0.04 (0.51)

That's...not very much code it seems to me.  (But I can imagine that verification is a nontrivial cost, again, not very much code here)

(In reply to Colin Walters from comment #23)
> A few options I see for this:
> 
> 1) Add glib's HMAC implementation to the validated list

Can you explain what does this mean?
We are not expanding the number of packages we are doing FIPS validation for, we are actually shrinking it by moving all crypto operations in the crypto libraries
FIPS validation is an extremely expensive process (both in terms of time and money), and would require you to do 20x the work you have now (you need to run self tests at runtime, and a bunch of other things).

> 2) Write patch to have glib's HMAC redirect to nss/openssl

Yes please.

> 3) Verify that no packages shipped in RHEL use this HMAC API, and *disable*
> it in FIPS mode

This is also fine.

> Any other variants I'm missing?
> 
> I can quickly try to gather some data on 3) as that seems clearly simplest. 
> Though of course none of these are mutually exclusive, we can start with 3)
> and go to 1) or 2) later.

If there are users you can also ask them to swith to openssl and still do 3, however this analysis is coming very late, devel freeze is next week and it would be their right to push back to you.

https://gitlab.gnome.org/GNOME/glib/merge_requests/897

$ cat ghmac.c
#include <glib.h>

int
main (int argc, char **argv)
{
    const char key[] = "42";
    const char val[] = "foo";
    GHmac *hmac = g_hmac_new (G_CHECKSUM_SHA1, key, sizeof (key));
    g_hmac_update (hmac, val, sizeof (val));
    g_print ("%s\n", g_hmac_get_string (hmac));
}
$ ./ghmac
322a4aa0856e64f2f1dc2b35b02ce249a9744b43
$ echo 1 > /tmp/one
$ sudo mount --bind /tmp/one /proc/sys/crypto/fips_enabled 
$ ./ghmac 
(process:695): GLib-ERROR **: 19:40:50.190: g_hmac_new() is not available in FIPS mode

I guess a variant of 2) is "detect FIPS mode and dlopen() openssl".  That would push back the problem of glib linking a crypto library to FIPS mode only.

I really have to emphasize the large scope of glib pulling in a crypto library.  A whole lot of stuff links to glib, from Firefox/Chrome, qemu, ostree, pacemaker, and last I looked VMWare Workstation, etc.
And my #1 concern here is openssl-deprecated vs openssl.
Today Chrome links BoringSSL statically because otherwise: https://bugzilla.redhat.com/show_bug.cgi?id=1448649

For the record, cockpit does *not* use GHMac in RHEL 8 (only in RHEL 7). See https://bugzilla.redhat.com/show_bug.cgi?id=1718074#c2 for details.

I verified at least that our build of qemu is not using GHmac; qemu has an internal crypto abstraction and RHEL8's build uses gcrypt:
http://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/qemu-kvm/4.0.0/3.module+el8.1.0+3265+26c4ed71/data/logs/x86_64/build.log

(In reply to Colin Walters from comment #30)
> I guess a variant of 2) is "detect FIPS mode and dlopen() openssl".  That
> would push back the problem of glib linking a crypto library to FIPS mode
> only.
> 
> I really have to emphasize the large scope of glib pulling in a crypto
> library.  A whole lot of stuff links to glib, from Firefox/Chrome, qemu,
> ostree, pacemaker, and last I looked VMWare Workstation, etc.
> And my #1 concern here is openssl-deprecated vs openssl.
> Today Chrome links BoringSSL statically because otherwise:
> https://bugzilla.redhat.com/show_bug.cgi?id=1448649

I know it is sometimes a problem to use a cryptolibrary from a library due to dependencies.
The best course of action is to not implement cyrptographic functions in glib.
Can't you simply deprecate upstream and compile out gmac in RHEL ?

Is any RHEL package actually using it?

If not, it would probably be better to simply patch it out in the RHEL packages so that people will see failures at build time instead of at runtime only in FIPS mode.

In general it is bad to have crypto implemented outside of crypto libraries even when not running in FIPS mode because these applications then are not following crypto-policy configurations.

For example we'll soon disable SHA-1 by default everywhere and glib would happily continue to allow it's use in non-FIPS mode, this is inconsistent with our promise to allow the dmin to control the crypto policies system wide and another reason why we ask package to use crypto-libraries where possible.

I mentioned above shotwell; that was imported into rhel-8 dist-git but looks like it was decided not to ship it.

Then, what we come down to here is the "not in RHEL" code, particularly proprietary code not turned up by the code search.  We can force *all* apps linking to glib to also link to a crypto library, or we can break just ones using GHmac.  Given the data so far I'm strongly leaning towards the latter.

So I'll be applying https://gitlab.gnome.org/GNOME/glib/merge_requests/897 to dist-git soon.  (Code review would be cool if anyone feels up for it...)

> If not, it would probably be better to simply patch it out in the RHEL packages so that people will see failures at build time instead of at runtime only in FIPS mode.

But that would break 3rd party apps using it even when not in FIPS mode.  I'm not so sure about doing that.  Again there's just so much glib-using code out there.  I didn't mention GStreamer before, think about people doing custom media streaming servers/devices; if they happen to use GHmac for something...

Maybe an incremental middle ground here (and one upstreamable likely) is to deprecate the API, so people at least see warnings.  Then we can revisit trying to disable it after people have hopefully been seeing compiler warnings for a while.

I took a look at the change, it looks "ok" as far as hacks go, however it will probbaly cause bugs if there is any application that really use it, because peolple will rightfully complain that you are denying "approved" HMACs in FIPS mode, and that is unexpected.
 I am ok breking only apps that use GHmac, but I would rather have them break (also) at compile time, and not just at runtime.

Perhaps we can come to a compromise where we disable GHmac in the header files, so rebuilds will flag it out with a build failure (maybe with a #define around it that can re-enable the API if *really* someone needs it), but we keep the actual symbol in the library with your patch to disable it in FIPS mode ?

Colin, the documentation for g_hmac_new() allows returning %NULL, and does so for non-OOM conditions. For example if a digest is not supported. In FIPS mode, the entire scheme (HMAC) is not supported ... so %NULL seems appropriate no?

> Colin, the documentation for g_hmac_new() allows returning %NULL

As I commented in the patch, that's true but...at least Cockpit's code didn't check for NULL and so it would just result in a segfault rather than having a useful error message by default.

The Vala-generated C code that Shotwell is using also doesn't check for NULL.

https://src.osci.redhat.com/rpms/glib2/pull-request/1

(In reply to Simo Sorce from comment #33)
> Is any RHEL package actually using it?

In bug 1637801, pidgin is about to get switched to use GLib's GHmac instead of its own hand rolled copy. That's also what Pidgin development snapshots do.

I don't think it's reasonable to just yank out the entire GHmac API from glib2 and expect existing GLib users to figure it out on their own.

Look you have a choice,
remove it, or link to an approved crypto library.

In the *very short* term, Colin's patch is somewhat acceptable, but it is not a long term fix, as it actually breaks any use of the library (and of depending applications) in FIPS mode, and does not conforms to crypto-policies in non-FIPS mode.

So it is a hack that is allowable only for 8.1 because we are short on time, it is not a solution past 8.1 and will need more work for 8.2 if you go with this one in 8.1 

If you can't remove the API, and there are actual RHEL users of the API, then the longer term solution is to link to a crypto library either directly or via dlopen() every time this API is used. I do not think it is in spirit to link only when in FIPS mode.

It is really unfortunate glib painted itself in a corner by implementing cryptographic functions, but we need to solve this issue properly, we can't live with hacks long term.

(In reply to Simo Sorce from comment #42)
> It is really unfortunate glib painted itself in a corner by implementing
> cryptographic functions, but we need to solve this issue properly, we can't
> live with hacks long term.

The fact remains that directly using any of those cryptographic libraries is a pain that lot of upstreams don't want to deal with. There are all kinds of things out there that were rolling their own HMAC implementation, and that's why GHmac was added to GLib.

It's not like people add random API to a library that can never ever break API/ABI ever for fun. :)

A cursory search brought up folks, pidgin, rest, shotwell, Telepathy (when it was still relevant). There's probably more. Some of these used to be a part of RHEL, some aren't today, but the fact is that there are real users of it out there.

See https://bugzilla.gnome.org/show_bug.cgi?id=652480 for the full discussion about why GHmac was added.

Anyway, I'd leave it to the GLib maintainers to figure out a way forward. I can imagine GLib adding an extension point that can be backed by GnuTls and OpenSSL via glib-networking and glib-openssl [1] respectively.

But saying that nobody can ever touch GHmac doesn't look like a realistic solution to me. We'd be stuck writing and maintaining downstream patches for random modules forever.

[1] https://gitlab.gnome.org/Archive/glib-openssl

Hi,

(In reply to Colin Walters from comment #15)
> I guess we could pick gnutls or nss to avoid this, but then we're forcing a
> big swath of apps to link against an extra crypto library.
why is that a problem? They're probably already loaded in some running program anyway, and the mappings should be shared between processes, so it doesn't really seem like that big of a deal?

Ok, so I am confused now.
Rishi and Ray tell me glib2 already links dynamically to cryptographic libraries, so why is it a problem to use also HMAc from either gnutls or openssl ?

the problem is glib is more than one library. g_hmac is in libglib, and the gnutls/opessl stuff is from plugins loaded by libgio.  the abstraction over dlopen() is done by libgmodule which libgio links to but libglib doesn't.

of course, most programs that use glib link against all three, so libglib could use weak symbols or equivalent I guess...but that's kind of ugly. I wish we'd just declare gnutls a winner here and link against it directly without extension points and dlopen etc, maybe with a build flag to disable linking to it for distros that really can't have the dependency.

> Rishi and Ray tell me glib2 already links dynamically to cryptographic libraries, so why is it a problem to use also HMAc from either gnutls or openssl ?

That's only if glib-networking is loaded; it isn't in all cases.  That said my primary concern was about using openssl...it's true that it's going to be very common for glib apps to also pull in glib-networking which uses gnutls (today...there is a patch to port it to openssl which brings back a can of worms but we can ignore that for now).

I will spend some time today investigating a --enable-hmac-gnutls build-time option for glib which replaces the ghmac API with static inlines that redirect to gnutls.

https://gitlab.gnome.org/GNOME/glib/merge_requests/903

https://src.fedoraproject.org/rpms/glib2/pull-request/3
https://src.osci.redhat.com/rpms/glib2/pull-request/2

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:3530

FYI: this was never upstreamed to Fedora, let alone glib upstream, so it won't be included in RHEL 9.

Michael,
requirements haven't changed in RHEL-9,
do you need a clone of this bug ?

A clone for RHEL 9 wouldn't hurt.

Looks like back before I joined Red Hat, I agreed this could be added to Fedora if the downstream patch was updated slightly (to use gnutls_hmac_copy() to implement g_hmac_copy(), rather than crashing if that function is called). So we could just do that.

Upstream glib is going to be harder, though.