Bug 1809799 - /usr/lib64/ceph/libceph-common.so.0 isn't CET enabled
Summary: /usr/lib64/ceph/libceph-common.so.0 isn't CET enabled
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: ceph
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Kaleb KEITHLEY
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: 1802674
TreeView+ depends on / blocked
 
Reported: 2020-03-03 21:49 UTC by H.J. Lu
Modified: 2020-05-22 21:48 UTC (History)
11 users (show)

Fixed In Version: ceph-15.1.0-3
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-03-10 14:25:44 UTC
Type: Bug


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Ceph Project Bug Tracker 44403 0 None None None 2020-03-03 22:04:16 UTC
Github intel isa-l pull 138 0 None closed x86: Add ENDBR32/ENDBR64 at function entries for Intel CET 2020-07-21 15:52:44 UTC
Github intel isa-l pull 139 0 None closed x86: Generate .note.gnu.property section for ELF output 2020-07-21 15:52:44 UTC

Description H.J. Lu 2020-03-03 21:49:44 UTC
[hjl@gnu-cfl-1 ~]$ readelf -n /usr/lib64/ceph/libceph-common.so.0 2>&1 | grep IBT
[hjl@gnu-cfl-1 ~]$

Comment 1 Kaleb KEITHLEY 2020-03-03 22:04:17 UTC
care to elaborate? Otherwise I have no idea what this is about.

Comment 2 H.J. Lu 2020-03-03 22:39:14 UTC
Intel Control-flow Enforcement Technology (CET):

https://software.intel.com/en-us/articles/intel-sdm

contains shadow stack (SHSTK) and indirect branch tracking (IBT).  When
CET is enabled, ELF object files must be marked with .note.gnu.property
section.  Also when IBT is enabled, all indirect branch targets must
start with ENDBR instruction.  CET is being enabled on Fedora:

https://bugzilla.redhat.com/show_bug.cgi?id=1538725

Comment 3 Kaleb KEITHLEY 2020-03-04 12:03:06 UTC
(In reply to H.J. Lu from comment #2)
> Intel Control-flow Enforcement Technology (CET):
> 
> https://software.intel.com/en-us/articles/intel-sdm
> 
> contains shadow stack (SHSTK) and indirect branch tracking (IBT).  When
> CET is enabled, ELF object files must be marked with .note.gnu.property
> section.  Also when IBT is enabled, all indirect branch targets must
> start with ENDBR instruction.  CET is being enabled on Fedora:
> 

Yes, I can google as well as anyone. You still haven't explained what the problem is. And if Fedora is enabling it in the RPM packaging bits then my presumption is that nothing further needs to be done.

I'm guessing, now after looking further, that you're trying to tell me that SHSTK and IBT are enabled in the other ceph libraries, but not in the libceph-common.so; is that what this is BZ is about?

Comment 4 H.J. Lu 2020-03-04 12:35:21 UTC
To enable CET, all linker inputs must be CET enabled, -Wl,-z,cet-report=error reports:

/usr/local/bin/ld: ../lib/libcrc32.a(crc32c_intel_fast_asm.s.o): error: missing IBT and SHSTK properties
/usr/local/bin/ld: ../lib/libcrc32.a(crc32c_intel_fast_zero_asm.s.o): error: missing IBT and SHSTK properties
/usr/local/bin/ld: ../boost/lib/libboost_thread.a(thread.o): error: missing IBT and SHSTK properties
/usr/local/bin/ld: ../boost/lib/libboost_thread.a(once.o): error: missing IBT and SHSTK properties
/usr/local/bin/ld: ../boost/lib/libboost_system.a(error_code.o): error: missing IBT and SHSTK properties
/usr/local/bin/ld: ../boost/lib/libboost_iostreams.a(zlib.o): error: missing IBT and SHSTK properties

Comment 5 Kaleb KEITHLEY 2020-03-10 14:25:44 UTC
fixed in ceph-15.1.0-3

some of this, i.e. the boost, will be eliminated when or if f33/rawhide updates to boost-1.71 and the ceph build can switch to building with the system's boost lib.

Comment 6 Kaleb KEITHLEY 2020-05-21 18:11:05 UTC
In retrospect, I think there's a fatal flaw here in how this is enforced.

Ceph has a pair of hand-written optimized assembly source files. (They're written in NASM/YASM format and gas can't be used to assemble them. This is compounded by yasm not being able to generate NOTE sections in the .o output file.)

So basically one or two .o files that don't have the requisite .note.gnu.property section (and content) will "taint" a whole shlib that otherwise consists of gcc or g++ or clang compiled .o files that do have the required .note.gnu.property section.

Ceph probably isn't the only project that has this issue. I have employed a hack to fix Ceph in Fedora; how are other projects solving this?

And I don't think anyone wants to rewrite the .s files to be able to use gas instead.

Comment 7 H.J. Lu 2020-05-21 18:49:25 UTC
Adding the CET marker is the part of CET enabling process.  There is a workaround:

1. Pass "-z cet-report=error" to linker to identify all input files which
don't have the CET marker.
2. Exam these input files to verify that the only thing missing for CET is the
CET marker.
3. Pass "-z ibt -z shstk" to linker to ignore missing CET marker in input files.

Comment 8 Kaleb KEITHLEY 2020-05-21 23:29:47 UTC
I already have a work-around: adding a .note.gnu.property note section with the shstk and ibt magic bits. Lying about it having shstk and ibt seems like at least as much of a hack as telling the linker to ignore the missing CET marker(s). If I just tell the linker to ignore it, aren't we going to be right back at square one when someone actually checks it again?

The particular file is https://github.com/intel/isa-l/blob/master/crc/crc32_iscsi_00.asm, which Ceph has bundled copies of.

I have opened an issue in github. I imagine that enabling CET in (what I presume is) highly optimized hand-coded assembly is likely to be non-trivial.

see https://github.com/intel/isa-l/issues/137

Comment 9 H.J. Lu 2020-05-21 23:45:43 UTC
(In reply to Kaleb KEITHLEY from comment #8)
> I have opened an issue in github. I imagine that enabling CET in (what I
> presume is) highly optimized hand-coded assembly is likely to be non-trivial.

Just add ENDBR and a CET marker.  It shouldn't be that hard.

Comment 10 Kaleb KEITHLEY 2020-05-22 12:48:38 UTC
(In reply to H.J. Lu from comment #9)
> (In reply to Kaleb KEITHLEY from comment #8)
> > I have opened an issue in github. I imagine that enabling CET in (what I
> > presume is) highly optimized hand-coded assembly is likely to be non-trivial.
> 
> Just add ENDBR and a CET marker.  It shouldn't be that hard.

Thanks. As I mentioned in comment #8, I've already added the CET marker to the Fedora package builds. Apart from that I prefer not to propose changes in the Ceph codebase that diverge from the upstream intel bits. I will relay your comment about adding ENDBR. If it's added upstream I'll be happy to pull any changes into Ceph.

Greg Tucker (a.k.a. gbtucker@github, one of the committers) writes:

  Generally ISA-L has no indirect branches in the asm code except in the
  function dispatcher. The dispatcher checks for cpu capability with an
  indirect call but it always matches the ret (shstk compliant). The
  dispatcher then calls function versions with an indirect jmp instruction.
  We don't start functions with an endbranch instruction so I'm not sure
  how this works for ibt.

Comment 11 H.J. Lu 2020-05-22 21:48:17 UTC
(In reply to Kaleb KEITHLEY from comment #10)
> (In reply to H.J. Lu from comment #9)
> > (In reply to Kaleb KEITHLEY from comment #8)
> > > I have opened an issue in github. I imagine that enabling CET in (what I
> > > presume is) highly optimized hand-coded assembly is likely to be non-trivial.
> > 
> > Just add ENDBR and a CET marker.  It shouldn't be that hard.
> 
> Thanks. As I mentioned in comment #8, I've already added the CET marker to
> the Fedora package builds. Apart from that I prefer not to propose changes
> in the Ceph codebase that diverge from the upstream intel bits. I will relay
> your comment about adding ENDBR. If it's added upstream I'll be happy to
> pull any changes into Ceph.
> 
> Greg Tucker (a.k.a. gbtucker@github, one of the committers) writes:
> 
>   Generally ISA-L has no indirect branches in the asm code except in the
>   function dispatcher. The dispatcher checks for cpu capability with an
>   indirect call but it always matches the ret (shstk compliant). The
>   dispatcher then calls function versions with an indirect jmp instruction.
>   We don't start functions with an endbranch instruction so I'm not sure
>   how this works for ibt.

Please try my patches.


Note You need to log in before you can comment on or make changes to this bug.