RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2121123 - Some binaries built with an excessive number of CIE entries in the Call Frame Information (CFI)
Summary: Some binaries built with an excessive number of CIE entries in the Call Frame...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: binutils
Version: 9.0
Hardware: All
OS: Linux
unspecified
low
Target Milestone: rc
: ---
Assignee: Nick Clifton
QA Contact: Miloš Prchlík
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-08-24 14:29 UTC by William Cohen
Modified: 2023-05-09 10:38 UTC (History)
4 users (show)

Fixed In Version: binutils-2.35.2-25.el9
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-05-09 08:24:17 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-132206 0 None None None 2022-08-24 14:30:58 UTC
Red Hat Product Errata RHBA-2023:2548 0 None None None 2023-05-09 08:24:31 UTC

Description William Cohen 2022-08-24 14:29:29 UTC 
Description of problem:

The Call Frame Information (CIE) for some installed binaries on RHEL9 rather than have one or a few Common Information Entries (CIEs) they have dozens or even hundreds of CIEs.  Initially noticed this problem on some of the installed shared libraries of Fedora 35.  The multiple CIE are not observed on Fedora Rawhide, but can be seen on RHEL9.  It looks like the issue is related to upstream binutils git commit  d20eb46617fc6170e144f80dcbaffbc3f4ff6090

The effect of this can be seen in the various shared libraries with following:

find /lib64/ -name "*.so.*" -type f | xargs -I % sh -c "echo -n '% '; readelf --debug-dump=frames % |grep "^0"| egrep 'CIE' |awk '{print \$4}' | sort |uniq -c | awk '{print \$1}' " |sort -nrk 2 > x86_64_rhel9_cie.log

more x86_64_rhel9_cie.log
/lib64/libwebkit2gtk-4.0.so.37.56.10 317
/lib64/libicui18n.so.67.1 302
/lib64/libjavascriptcoregtk-4.0.so.18.20.10 249
/lib64/dyninst/libparseAPI.so.12.1.0 214
/lib64/dyninst/libdyninstAPI.so.12.1.0 198
/lib64/libexempi.so.8.1.0 190
/lib64/libexiv2.so.0.27.5 185
/lib64/libpython3.9.so.1.0 181
/lib64/libpoppler.so.106.0.0 180
/lib64/libqpdf.so.28.3.1 170
/lib64/libdnf.so.2 170
/lib64/libicuuc.so.67.1 157
/lib64/libgjs.so.0.0.0 141
...

Version-Release number of selected component (if applicable):
binutils-2.35.2-24.el9.x86_64

How reproducible:

Every time


Steps to Reproduce:

# set up CodeReady builder for rhel9 or do this on CentOS Stream 9
# then use the following demonstrate the issue

0. sudo dnf install rpm-build wget
1. wget https://kojipkgs.fedoraproject.org//packages/dyninst/12.1.0/4.fc37/src/dyninst-12.1.0-4.fc37.src.rpm
2. sudo dnf builddep dyninst-12.1.0-4.fc37.src.rpm
3. rpm -Uvh dyninst-12.1.0-4.fc37.src.rpm
4. cd rpmbuild/SPECS
5. rpmbuild -ba dyninst.spec
6. cd ~/rpmbuild/BUILD/dyninst*
7. find ./ -name "*.so.*" -type f | xargs -I % sh -c "echo -n '% '; readelf --debug-dump=frames % |grep "^0"| egrep 'CIE' |awk '{print \$4}' | sort |uniq -c | awk '{print \$1}' " |sort -nrk 2 

Actual results:

$ find ./ -name "*.so.*" -type f | xargs -I % sh -c "echo -n '% '; readelf --debug-dump=frames % |grep "^0"| egrep 'CIE' |awk '{print \$4}' | sort |uniq -c | awk '{print \$1}' " |sort -nrk 2
./install/usr/lib64/dyninst/libparseAPI.so.12.1.0 214
./dyninst-12.1.0/redhat-linux-build/parseAPI/libparseAPI.so.12.1.0 214
./install/usr/lib64/dyninst/libdyninstAPI.so.12.1.0 198
./dyninst-12.1.0/redhat-linux-build/dyninstAPI/libdyninstAPI.so.12.1.0 198
./install/usr/lib64/dyninst/libpcontrol.so.12.1.0 121
./dyninst-12.1.0/redhat-linux-build/proccontrol/libpcontrol.so.12.1.0 121
./install/usr/lib64/dyninst/libsymtabAPI.so.12.1.0 111
./dyninst-12.1.0/redhat-linux-build/symtabAPI/libsymtabAPI.so.12.1.0 111
./install/usr/lib64/dyninst/libinstructionAPI.so.12.1.0 87
./dyninst-12.1.0/redhat-linux-build/instructionAPI/libinstructionAPI.so.12.1.0 87
./install/usr/lib64/dyninst/libcommon.so.12.1.0 67
./dyninst-12.1.0/redhat-linux-build/common/libcommon.so.12.1.0 67
./install/usr/lib64/dyninst/libpatchAPI.so.12.1.0 53
./dyninst-12.1.0/redhat-linux-build/patchAPI/libpatchAPI.so.12.1.0 53
./install/usr/lib64/dyninst/libdynDwarf.so.12.1.0 23
./dyninst-12.1.0/redhat-linux-build/dwarf/libdynDwarf.so.12.1.0 23
./install/usr/lib64/dyninst/libdynC_API.so.12.1.0 18
./dyninst-12.1.0/redhat-linux-build/dynC_API/libdynC_API.so.12.1.0 18
./install/usr/lib64/dyninst/libstackwalk.so.12.1.0 17
./dyninst-12.1.0/redhat-linux-build/stackwalk/libstackwalk.so.12.1.0 17
./install/usr/lib64/dyninst/libdynElf.so.12.1.0 15
./dyninst-12.1.0/redhat-linux-build/elf/libdynElf.so.12.1.0 15
./install/usr/lib64/dyninst/libsymLite.so.12.1.0 8
./dyninst-12.1.0/redhat-linux-build/symlite/libsymLite.so.12.1.0 8
./install/usr/lib64/dyninst/libdyninstAPI_RT.so.12.1.0 1
./dyninst-12.1.0/redhat-linux-build/dyninstAPI_RT/libdyninstAPI_RT.so.12.1.0 1


Expected results:

On x86_64 Fedora rawhide see that the number of CIE entries are relatively few in each of the libraries:
$ find ./ -name "*.so.*" -type f | xargs -I % sh -c "echo -n '% '; readelf --debug-dump=frames % |grep "^0"| egrep 'CIE' |awk '{print \$4}' | sort |uniq -c | awk '{print \$1}' " |sort -nrk 2
./install/usr/lib64/dyninst/libsymtabAPI.so.12.1.0 3
./install/usr/lib64/dyninst/libsymLite.so.12.1.0 3
./install/usr/lib64/dyninst/libstackwalk.so.12.1.0 3
./install/usr/lib64/dyninst/libpcontrol.so.12.1.0 3
./install/usr/lib64/dyninst/libpatchAPI.so.12.1.0 3
./install/usr/lib64/dyninst/libparseAPI.so.12.1.0 3
./install/usr/lib64/dyninst/libinstructionAPI.so.12.1.0 3
./install/usr/lib64/dyninst/libdyninstAPI.so.12.1.0 3
./install/usr/lib64/dyninst/libdynElf.so.12.1.0 3
./install/usr/lib64/dyninst/libdynDwarf.so.12.1.0 3
./install/usr/lib64/dyninst/libdynC_API.so.12.1.0 3
./install/usr/lib64/dyninst/libcommon.so.12.1.0 3
./dyninst-12.1.0/redhat-linux-build/symtabAPI/libsymtabAPI.so.12.1.0 3
./dyninst-12.1.0/redhat-linux-build/symlite/libsymLite.so.12.1.0 3
./dyninst-12.1.0/redhat-linux-build/stackwalk/libstackwalk.so.12.1.0 3
./dyninst-12.1.0/redhat-linux-build/proccontrol/libpcontrol.so.12.1.0 3
./dyninst-12.1.0/redhat-linux-build/patchAPI/libpatchAPI.so.12.1.0 3
./dyninst-12.1.0/redhat-linux-build/parseAPI/libparseAPI.so.12.1.0 3
./dyninst-12.1.0/redhat-linux-build/instructionAPI/libinstructionAPI.so.12.1.0 3
./dyninst-12.1.0/redhat-linux-build/elf/libdynElf.so.12.1.0 3
./dyninst-12.1.0/redhat-linux-build/dyninstAPI/libdyninstAPI.so.12.1.0 3
./dyninst-12.1.0/redhat-linux-build/dynC_API/libdynC_API.so.12.1.0 3
./dyninst-12.1.0/redhat-linux-build/dwarf/libdynDwarf.so.12.1.0 3
./dyninst-12.1.0/redhat-linux-build/common/libcommon.so.12.1.0 3
./install/usr/lib64/dyninst/libdyninstAPI_RT.so.12.1.0 1
./dyninst-12.1.0/redhat-linux-build/dyninstAPI_RT/libdyninstAPI_RT.so.12.1.0 1

Additional info:

RHEL 8 suffers has high CIE counts for it shared libraries.
Fedora 36 has low CIE counts, so binutils-2.37-27.fc36.x86_64 also addresses the problem.
Comment 1 Nick Clifton 2022-08-24 16:05:46 UTC 
(In reply to William Cohen from comment #0)

Hi Will,

> The Call Frame Information (CIE) for some installed binaries on RHEL9 rather
> than have one or a few Common Information Entries (CIEs) they have dozens or
> even hundreds of CIEs.  

Why does this matter ?

The debug information is correct, yes ?


> It looks like the issue is related to upstream binutils git commit  
> d20eb46617fc6170e144f80dcbaffbc3f4ff6090
...
> RHEL 8 suffers has high CIE counts for it shared libraries.
> Fedora 36 has low CIE counts, so binutils-2.37-27.fc36.x86_64 also addresses the problem.

Interestingly F36 binutils does include commit d20eb46617fc6170e144f80dcbaffbc3f4ff6090
whereas RHEL 8 (which is based upon binutils 2.30) does not.  Plus the RHEL 8 binutils is
very old now.  So possibly this is a "feature" of the RHEL build system rather than a 
specific binutils patch.
Comment 2 William Cohen 2022-08-24 17:49:43 UTC 
The CIE information appears correct, but the additional CIEs take up additional space.  It isn't a huge amount of space (on the order of 20-30 bytes per CIE).  Thus, the bug is marked as low priority as it is relatively minor. Yes, RHEL 8 is getting old and I didn't think it was worth it to file a this issue for RHEL8.  I was debating whether to file for RHEL9. :)

The reproducer was to show problem exists for locally built binaries too. The reproducer works on a rpmbuild and the examination was on the binaries in the build directory.  This avoids some of the other build system things like striping out the debuginfo and dwz compression operation.

Yes, fixing binutils is not going to change the system binaries unless they are built with the patched binutils.  I wouldn't suggest rebuilding all the existing RHEL binaries to make them smaller.  However, future RHEL9.2 builds and external customer application built with the patched binutils could a bit more compact.  Given the CFI info is always in binaries it might also make container images a bit smaller.
Comment 3 Nick Clifton 2022-08-25 15:50:45 UTC 
Just to confirm:  This problem appears to be specific to the x86_64 architecture, and specific to RHEL:

Count of CIE entries in libparseAPI.so.12.1.0:

  Architecture: AArch64  i686  PowerPC  x86_64
  --------------------------------------------
  RHEL-8:          3       6      3       135
  RHEL-9:          4      14      3       214
  F37:             3       2      3         3

It seems that the i686 architecture is also affected, but to a much smaller degree.

The problem is that I have no idea what is causing this effect.  I would like to blame the compiler, since it generates the cfi directives that are consumed by the assembler to make the contents of the .eh_frame section.  But I do not have any way to justify this at the moment.

Will - I am not familiar with the .eh_frame section's contents.  Can you tell me what these excess CIEs actually do ?  Are they associated with missing pieces of code for example, or duplicates of other entries ?
Comment 4 William Cohen 2022-08-25 16:23:31 UTC 
The CIEs are Common Information Entries used in the Call Frame Information (CFI) for unwinding the stack.  The CIE are used as the starting default for the rules for unwinding the stack.  Each function has a Frame Description Entry (FDE) that describes how to unwind the state back to the previous call at each instruction in the function.  The FDE points to a CIE that describes the starting state of the registers/stack for unwinding.

https://refspecs.linuxfoundation.org/LSB_3.0.0/LSB-PDA/LSB-PDA/ehframechpt.html

The main purpose of the CIE is to factor out common information between the multiple FDE.  However, in the x86_64 case it seems to be doing something to cause a lot of unique CIEs to be created. That kind of defeats the purpose of factoring out the CIE if there are many variant CIE built.  As mentioned it isn't causing functional problems. just taking up additional space.

I noticed the problem on x86_64 f35 also.  One good test I didn't do would be to use the f37 (or f36) binutils on f35.
Comment 5 William Cohen 2022-08-29 17:52:03 UTC 
Tried out locally built versions of fedora binutils and using that the build the reproducer. Keeping the compiler and other packages on the system the same and just changing binutils.  Found the multiple CIEs in the reproducer are seen with binutils-2.36.1-15.fc35, but not with binutils-2.37-10.fc35.

To gauge the had two versions of the dyinst builds.  One ~/rpmbuild/dynsint-12.1.0.sysbinutils was built with RHEL9 binutils and the other ~/rpmbuild/dynsint-12.1.0.newbinutils was built with binutils-2.37-25.  Took a look at the file that had to most CIEs.

cd ~/rpmbuild/BUILD/dyninst-12.1.0.sysbinutils/
readelf -S ./dyninst-12.1.0/redhat-linux-build/parseAPI/libparseAPI.so.12.1.0 > ~/sections.sysbinutils.log
cd ../dyninst-12.1.0.newbinutils/
readelf -S ./dyninst-12.1.0/redhat-linux-build/parseAPI/libparseAPI.so.12.1.0 > ~/sections.newbinutils.log
diff -u ~/sections.sysbinutils.log ~/sections.newbinutils.log 

The portion of interest from the diff is:

@@ -38,50 +38,50 @@
   [16] .eh_frame_hdr     PROGBITS         00000000002a8310  002a8310
        0000000000009e5c  0000000000000000   A       0     0     4
   [17] .eh_frame         PROGBITS         00000000002b2170  002b2170
-       0000000000038d1c  0000000000000000   A       0     0     8
-  [18] .gcc_except_table PROGBITS         00000000002eae8c  002eae8c
+       0000000000039e04  0000000000000000   A       0     0     8
+  [18] .gcc_except_table PROGBITS         00000000002ebf74  002ebf74
        00000000000211ec  0000000000000000   A       0     0     4

Hmmm, the eh_frame with newer binutils is about 0x1e8, (4328 bytes larger).

Looking at the next highest CIE count, ./install/usr/lib64/dyninst/libdyninstAPI.so.12.1.0:

@@ -38,17 +38,17 @@
   [16] .eh_frame_hdr     PROGBITS         00000000001c4060  001c4060
        0000000000006ea4  0000000000000000   A       0     0     4
   [17] .eh_frame         PROGBITS         00000000001caf08  001caf08
-       0000000000029bac  0000000000000000   A       0     0     8
-  [18] .gcc_except_table PROGBITS         00000000001f4ab4  001f4ab4
+       0000000000029738  0000000000000000   A       0     0     8
+  [18] .gcc_except_table PROGBITS         00000000001f4640  001f4640
        000000000000b354  0000000000000000   A       0     0     4

There is some space savings, 0x474, 1140 bytes.  

For ./install/usr/lib64/dyninst/libpcontrol.so.12.1.0 looks like the new binutils generates an .eh_frame section that is about 1K larger:

@@ -38,48 +38,48 @@
   [16] .eh_frame_hdr     PROGBITS         00000000001291a8  001291a8
        00000000000056bc  0000000000000000   A       0     0     4
   [17] .eh_frame         PROGBITS         000000000012e868  0012e868
-       000000000001c4c8  0000000000000000   A       0     0     8
-  [18] .gcc_except_table PROGBITS         000000000014ad30  0014ad30
+       000000000001c8a8  0000000000000000   A       0     0     8
+  [18] .gcc_except_table PROGBITS         000000000014b110  0014b110
        000000000000821c  0000000000000000   A       0     0     4

It looksing like there might not be consistent space savings by factoring out the CIE information.
Comment 6 Nick Clifton 2022-09-08 15:36:44 UTC 
Hi Will,

Right - I have finally managed to get my head around this problem.

You are right - the commit referenced in the description is the solution that is needed.
I have a local build with the patch installed, but currently there are problems with the Zuul build system, so the patch cannot be merged into the RHEL-9 sources.  This problem is under investigation.

Cheers
  Nick
Comment 7 Nick Clifton 2022-09-12 14:47:05 UTC 
Fixed in binutils-2.35.2-25.el9
Comment 14 Miloš Prchlík 2023-01-09 08:20:56 UTC 
Verified with binutils-2.35.2-33.el9.
Comment 16 errata-xmlrpc 2023-05-09 08:24:17 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (binutils bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:2548
Note You need to log in before you can comment on or make changes to this bug.