Created attachment 1754468 [details]
WIP patch

I've hacked up a WIP patch that gets rid of all of the SSP failures in early startup code.  As promised earlier, __libc_start_main gets SSP while others are dropped from annobin since they're too early.

There still are failures remaining but they're all in ifuncs.  I've spoken to Nick Clifton and he has agreed to look at getting annobin to skip over ifuncs since they too are unsuitable for stack protector.

Failures with the patch:

Hardened: ./lib64/libc-2.32.9000.so: FAIL: (component: strcat): No stack protection enabled.
Hardened: ./lib64/libc-2.32.9000.so: FAIL: (component: strchr): No stack protection enabled.
Hardened: ./lib64/libc-2.32.9000.so: FAIL: Parts of the binary were compiled without a suffcient -fstack-protector setting.
Hardened: ./lib64/libmvec-2.32.9000.so: FAIL: (component: _ZGVbN2v_cos): No stack protection enabled.
Hardened: ./lib64/libmvec-2.32.9000.so: FAIL: (component: _ZGVeN8v_cos): No stack protection enabled.
Hardened: ./lib64/libmvec-2.32.9000.so: FAIL: Parts of the binary were compiled without a suffcient -fstack-protector setting.
Hardened: ./lib64/libm-2.32.9000.so: FAIL: (component: __asin_finite): No stack protection enabled.
Hardened: ./lib64/libm-2.32.9000.so: FAIL: (component: cosf): No stack protection enabled.
Hardened: ./lib64/libm-2.32.9000.so: FAIL: Parts of the binary were compiled without a suffcient -fstack-protector setting.
Hardened: ./sbin/ldconfig: FAIL: (component: strchr_ifunc): No stack protection enabled.
Hardened: ./sbin/ldconfig: FAIL: (component: strcmp_ifunc): No stack protection enabled.
Hardened: ./sbin/ldconfig: FAIL: Parts of the binary were compiled without a suffcient -fstack-protector setting.

As mentioned earlier, they are all ifuncs.

(In reply to Siddhesh Poyarekar from comment #2)
> Failures with the patch:
> 
> Hardened: ./lib64/libc-2.32.9000.so: FAIL: (component: strcat): No stack
> protection enabled.
> Hardened: ./lib64/libc-2.32.9000.so: FAIL: (component: strchr): No stack
> protection enabled.
> Hardened: ./lib64/libc-2.32.9000.so: FAIL: Parts of the binary were compiled
> without a suffcient -fstack-protector setting.
> Hardened: ./lib64/libmvec-2.32.9000.so: FAIL: (component: _ZGVbN2v_cos): No
> stack protection enabled.
> Hardened: ./lib64/libmvec-2.32.9000.so: FAIL: (component: _ZGVeN8v_cos): No
> stack protection enabled.
> Hardened: ./lib64/libmvec-2.32.9000.so: FAIL: Parts of the binary were
> compiled without a suffcient -fstack-protector setting.
> Hardened: ./lib64/libm-2.32.9000.so: FAIL: (component: __asin_finite): No
> stack protection enabled.
> Hardened: ./lib64/libm-2.32.9000.so: FAIL: (component: cosf): No stack
> protection enabled.
> Hardened: ./lib64/libm-2.32.9000.so: FAIL: Parts of the binary were compiled
> without a suffcient -fstack-protector setting.
> Hardened: ./sbin/ldconfig: FAIL: (component: strchr_ifunc): No stack
> protection enabled.
> Hardened: ./sbin/ldconfig: FAIL: (component: strcmp_ifunc): No stack
> protection enabled.
> Hardened: ./sbin/ldconfig: FAIL: Parts of the binary were compiled without a
> suffcient -fstack-protector setting.
> 
> As mentioned earlier, they are all ifuncs.

I want to clarify here because we have IFUNC resolver functions and the implementations.

The IFUNC resolver functions can't be easily protected right now, so we need a way to exclude them.

Are these all resolvers?

The implementations returned by the resolvers should be protected or marked as OK if implemented in assembly.

(In reply to Carlos O'Donell from comment #3)
> I want to clarify here because we have IFUNC resolver functions and the
> implementations.
> 
> The IFUNC resolver functions can't be easily protected right now, so we need
> a way to exclude them.
> 
> Are these all resolvers?

Yes they're all STT_GNU_IFUNC.  Nick is working on skipping them in annobin.

This bug appears to have been reported against 'rawhide' during the Fedora 34 development cycle.
Changing version to 34.

Jan, Siddhesh:

  I have a local version of annocheck with a few updates which now produces these results:

   % rpm -ql glibc | xargs annocheck -v --ignore-gaps --skip-all --test-pic --test-pie --test-stack-prot | grep FAIL
   Hardened: /sbin/ldconfig: FAIL: stack-prot test because no protection enabled (function: get_common_indices.constprop.0) 
   Hardened: /sbin/ldconfig: FAIL: stack-prot test because no protection enabled (function: check_one_fd) 
   Hardened: /lib64/ld-2.31.so: FAIL: stack-prot test because no protection enabled (function: is_dst) 
   annocheck: Warning: '/etc/gai.conf': No such file.

  Is this sufficient ?

  The updates include skipping tests for ifuncs, as well as adding special case exceptions for a few well known glibc startup functions (such as _dl_start, __libc_start_main and __libc_init_first).  I could add exceptions for is_dst, check_one_fd, and get_common_indicies but I am not sure if this is a good idea as these function names might be used in other packages.

Cheers
  Nick

(In reply to Nick Clifton from comment #7)
> Jan, Siddhesh:
> 
>   I have a local version of annocheck with a few updates which now produces
> these results:
> 
>    % rpm -ql glibc | xargs annocheck -v --ignore-gaps --skip-all --test-pic
> --test-pie --test-stack-prot | grep FAIL
>    Hardened: /sbin/ldconfig: FAIL: stack-prot test because no protection
> enabled (function: get_common_indices.constprop.0) 
>    Hardened: /sbin/ldconfig: FAIL: stack-prot test because no protection
> enabled (function: check_one_fd) 
>    Hardened: /lib64/ld-2.31.so: FAIL: stack-prot test because no protection
> enabled (function: is_dst) 
>    annocheck: Warning: '/etc/gai.conf': No such file.

All of them are early startup and could be added to exceptions.

>   Is this sufficient ?
> 
>   The updates include skipping tests for ifuncs, as well as adding special
> case exceptions for a few well known glibc startup functions (such as
> _dl_start, __libc_start_main and __libc_init_first).  I could add exceptions
> for is_dst, check_one_fd, and get_common_indicies but I am not sure if this
> is a good idea as these function names might be used in other packages.

This is great because I don't have to hack in annobin exclusion flags in glibc package build; upstream has been resistant to the idea of exposing internal flag options that were necessary to enable this.

I wonder though, if it would be easier for annobin to record -fno-stack-protector and use that to skip the stack protector tests.  That way, only objects that are explicitly built with -fno-stack-protector would be excluded; we do this in glibc and any reported failures then would be a hint for us to build those objects with either -fno-stack-protector or -fstack-protector.

(In reply to Siddhesh Poyarekar from comment #8)

> I wonder though, if it would be easier for annobin to record
> -fno-stack-protector and use that to skip the stack protector tests.

Annobin does record -no-stack-protector, but currently annocheck considers
this to be a FAIL result.  The issue is that for normal packages compiling
with -fno-stack-protector would surely be a bad idea.

> That
> way, only objects that are explicitly built with -fno-stack-protector would
> be excluded; we do this in glibc and any reported failures then would be a
> hint for us to build those objects with either -fno-stack-protector or
> -fstack-protector.

OK, so you are saying that for any package, compiling with -fno-stack-protector
indicates a deliberate decision on the part of the package maintainer to remove
stack protection.  (And presumably that they know what they are doing when they
disable this feature).  I can see that.  OK, I will make this change.

Please hold the horses. :-)

We'd like to depend on annocheck to be able to check that (ideally) all components in the distribution used the standard compiler flags as configured in rpm macros. One of the ways how a component will not be compiled with the distro-standard flags is when somewhere between .spec's %build, cmake, make, and myriad of other mechanisms leading towards compiler invocation, the rpm distro flags will be overridden by upstream's default values. It's not just package maintainer's deliberate action that can make the change and put the -fno-stack-protector in, it's unintended situation that we'd like annocheck to catch.

Besides, do maintainers ever have a reason to use -fno-stack-protector? I understand the early startup code of glibc is special but why would generic packages use -fno-stack-protector?

Asking Steve Grubb to chime in in case I'm missing some aspect of this.

(In reply to Jan Pazdziora from comment #10)
> Please hold the horses. :-)
> 
> We'd like to depend on annocheck to be able to check that (ideally) all
> components in the distribution used the standard compiler flags as
> configured in rpm macros. One of the ways how a component will not be
> compiled with the distro-standard flags is when somewhere between .spec's
> %build, cmake, make, and myriad of other mechanisms leading towards compiler
> invocation, the rpm distro flags will be overridden by upstream's default
> values. It's not just package maintainer's deliberate action that can make
> the change and put the -fno-stack-protector in, it's unintended situation
> that we'd like annocheck to catch.

I guess the difference is between not specifying -fstack-protector and explicitly specifying -fno-stack-protector-string.  The possibility of RPM distro flags being overridden by upstream defaults could result in the -fstack-protector-strong being dropped, but could there be a situation where upstream defaults (or any other build mechanism) would accidentally *include* -fno-stack-protector?

> Besides, do maintainers ever have a reason to use -fno-stack-protector? I
> understand the early startup code of glibc is special but why would generic
> packages use -fno-stack-protector?

I suppose the possibility is rare; jemalloc (specifically, programs that statically link to it since jemalloc itself is not in RHEL-9) for example could have code running before full libc initialization is complete.

Perhaps there's a middle ground here where annobin WARNs when it sees -fno-stack-protector'd objects and it has to be explicitly waived by the maintainer.

Googling for

  site:github.com fno-stack-protector

shows that it is used in some way by nontrivial number of projects. That of course does not say anything about packages in Fedora and RHEL but the changes are non-zero.

(In reply to Siddhesh Poyarekar from comment #11)
> I suppose the possibility is rare; jemalloc (specifically, programs that
> statically link to it since jemalloc itself is not in RHEL-9) for example
> could have code running before full libc initialization is complete.

I took a closer look at this and even with something like jemalloc, the possibility of a non-glibc function being called before stack guard setup is remote.  So for all practical purposes we can consider only glibc to absolutely need to build some objects without stack-protector.

(In reply to Jan Pazdziora from comment #12)
> Googling for
> 
>   site:github.com fno-stack-protector
> 
> shows that it is used in some way by nontrivial number of projects. That of
> course does not say anything about packages in Fedora and RHEL but the
> changes are non-zero.

Hmm, in that case I suppose it makes more sense for us to add only glibc-specific exceptions like Nick mentioned in comment 7.

We have a bug open for avahi because it overrides the distro flags and does not get the stack protection we expect. So, I think glibc specific exclusions is best.

OK, the exceptions for glibc have been added to annobin-9.63-1.fc34

I've now tried the same command on fresh Fedora rawhide with glibc-2.33.9000-2.fc35.x86_64 and annobin-annocheck-9.70-1.fc35.x86_64 and the stack protector issues are gone, thanks.

However, annocheck also reports

Hardened: Warning: /sbin/ldconfig: Corrupt annobin note : end address == -1.
Hardened: Warning: /sbin/ldconfig: Corrupt annobin note : end address == -1.
Hardened: Warning: /sbin/ldconfig: Corrupt annobin note : end address == -1.
Hardened: Warning: /sbin/ldconfig: Corrupt annobin note : end address == -1.
Hardened: Warning: /sbin/ldconfig: Corrupt annobin note : end address == -1.
Hardened: Warning: /sbin/ldconfig: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libutil-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libutil-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libutil-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libthread_db-1.0.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libthread_db-1.0.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libthread_db-1.0.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/librt-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/librt-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/librt-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/librt-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libresolv-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libresolv-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libresolv-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libresolv-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libpthread-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libpthread-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libpthread-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libpthread-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libnss_files-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libnss_files-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libnss_files-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libnss_dns-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libnss_dns-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libnss_dns-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libnss_dns-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libnss_dns-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libnss_compat-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libnss_compat-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libnss_compat-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libmvec-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libmvec-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libmvec-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libm-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libm-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libm-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libdl-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libdl-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libdl-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libdl-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libc-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libc-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libc-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libc-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libc-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libc-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libc-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libc-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libc-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libc-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libc-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libc-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libc-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libc-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libc-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libanl-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libanl-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libanl-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libSegFault.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libSegFault.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libSegFault.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libSegFault.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libBrokenLocale-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libBrokenLocale-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/libBrokenLocale-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/ld-2.33.9000.so: Corrupt annobin note : end address == -1.
Hardened: Warning: /lib64/ld-2.33.9000.so: Corrupt annobin note : end address == -1.

Is that of concern?

(In reply to Jan Pazdziora from comment #17)
> I've now tried the same command on fresh Fedora rawhide with
> glibc-2.33.9000-2.fc35.x86_64 and annobin-annocheck-9.70-1.fc35.x86_64 and
> the stack protector issues are gone, thanks.
> 
> However, annocheck also reports
> 
> Hardened: Warning: /sbin/ldconfig: Corrupt annobin note : end address == -1.
[snip]

> Is that of concern?

Yes.  If the notes are corrupt then annocheck's analysis will be incomplete at best, and quite possibly wrong.

That said, I have just run annocheck on glibc-2.33.9000-4.fc35.x86_64.rpm and none of these errors showed up.  So possibly there was a build problem with .9000-2 glibc that is now fixed with the .9000-4 build, or, more likely, there is a bug in the version of annocheck that you used, which is fixed in the version that I used.  (I used annocheck version 9.70).

Nod, I confirm that upgrading glibc from 2.33.9000-2.fc35.x86_64 to 2.33.9000-4.fc35.x86_64 made the corrupt annobin note warnings go away and everything passes there. That's with annobin-annocheck-9.70-1.fc35.x86_64.

I thought I'd test the latest Fedora rawhide glibc build which at this time is glibc-2.33.9000-6.fc35 -- https://koji.fedoraproject.org/koji/buildinfo?buildID=1746208. However, the Fedora rawhide currently only ships glibc-2.33.9000-2.fc35 which is a build from February and I see multiple annocheck FAILs on non-x86_64 platforms there.

Looking at https://koji.fedoraproject.org/koji/buildinfo?buildID=1746208 it was built/tagged into the f35-build-side-40742 tag. Is there an easy way to enable this side tag as a dnf repository so that I could easily upgrade to these versions in automated fashion, without specifying all the koji locations manually?

We are getting close to importing glibc snapshots into rawhide proper again. See the announcement here:

Bringing glibc 2.34 snapshots to Fedora 35 and CentOS Stream 9
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/7R4RBS2MD75S2MTY42L44IW3KOUT5K36/

This message is a reminder that Fedora Linux 34 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 34 on 2022-06-07.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '34'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 34 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Fedora Linux 34 entered end-of-life (EOL) status on 2022-06-07.

Fedora Linux 34 is no longer maintained, which means that it
will not receive any further security or bug fix updates. As a result we
are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release.

Thank you for reporting this bug and we are sorry it could not be fixed.