Bug 833311

Summary: rpm dwz support
Product: [Fedora] Fedora Reporter: Jakub Jelinek <jakub>
Component: rpmAssignee: Fedora Packaging Toolset Team <packaging-team>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: ffesti, jnovy, mjw, packaging-team, pknirsch, pmatilai
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-08-02 06:55:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
find-debuginfo.sh patch
none
find-debuginfo.sh patch
none
rpm-dwz.patch
none
rpm-dwz.patch
none
redhat-rpm-config-9.1.0-dwz.patch none

Description Jakub Jelinek 2012-06-19 08:29:52 UTC 
Created attachment 592863 [details]
find-debuginfo.sh patch

The https://fedoraproject.org/wiki/Features/DwarfCompressor feature has been approved yesterday.
The dwz utility is already included in rawhide, binutils minimal support for readelf and objdump is building right now, gdb support is written and will hopefully be soon included.

But we need changes on the rpm and redhat-rpm-config side to enable this.

I'd like to see something like the following patch to find-debuginfo.sh (lightly tested so far), which will automatically run the dwz utility on the *.debug files
creates by stripping to file earlier in the script.

In addition to that I'd appreciate if -g3 could be enabled by default in redhat-rpm-config (something like):

--- macros.jj	2010-02-02 10:24:04.000000000 +0100
+++ macros	2012-06-19 09:59:45.647230617 +0200
@@ -177,7 +177,7 @@ unset DISPLAY\
 %__find_provides        /usr/lib/rpm/redhat/find-provides
 %__find_requires        /usr/lib/rpm/redhat/find-requires
 
-%__global_cflags	-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4
+%__global_cflags	-O2 -g3 -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4
 
 # Use these macros to differentiate between RH and other KMP implementation(s).
 redhat_kernel_module_package	1

and some way how to pass smaller or larger than default values of
--dwz-low-mem-die-limit 10000000 and --dwz-max-die-limit 50000000 parameters to find-debuginfo.sh.  The numbers depend on the minimum amount of RAM in tbe buildboxes and possibly CPU speed as well, for most of the packages the limits aren't reached, but there are problematic packages with huge amounts of debuginfo - libreoffice has ~ 2.5GB of debuginfo, but nothing really huge,
on x86_64 fits into ~ 2GB of RAM or so and on i?86 even less, but then there is libxul.so.debug and libwebkit*so*debug (in several packages ~ .5GB file, in several packages ~ 1.2GB file) on which with huge both limits it could need up to 10GB of RAM when not over low mem die limit and 5GB of RAM when over low mem limit, but below max die limit.  webkit has ~ 125million DIEs or so if I remember well.  On 32-bit hosts we have the problem that there is some small hard virtual address space limit.  The default limits 10mil and 50mil should be reasonable for i386, on x86_64 if buildboxes have say 4GB or better 6GB of RAM available, --dwz-max-die-limit could be increased say up to 150mil.
On ARM both the limits should be definitely lowered, both because the VA limit is 2GB, boxes have .5GB of RAM? and because it is terribly slow.

Any suggestions how to define per-architecture limits and pass them down into
_find_debuginfo_opts ?
Comment 1 Jakub Jelinek 2012-06-19 09:37:45 UTC 
Created attachment 592888 [details]
find-debuginfo.sh patch

Updated find-debuginfo.sh patch that only enables running dwz if --run-dwz option is given.
Comment 2 Jakub Jelinek 2012-06-19 10:39:27 UTC 
Created attachment 592903 [details]
rpm-dwz.patch

Yet another update of the patch, this time to create /usr/lib/debug/.build-id/XX/YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY.debug symlinks to ../../.dwz/%{nvra}.debug files.
Comment 3 Jakub Jelinek 2012-06-22 07:08:00 UTC 
Created attachment 593658 [details]
rpm-dwz.patch

As discussed on IRC, this patch just adds %{?_find_debuginfo_dwz_opts}
to find-debuginfo.sh arguments, so that redhat-rpm-config can enable it.
Comment 4 Jakub Jelinek 2012-06-22 07:19:06 UTC 
I've tried to come up with /etc/rpm/macros.dwz for redhat-rpm-config, but haven't succeeded so far, my rpm macro skills aren't sufficient:
=======================
# Macros for reducing debug info size using dwz(1) utility.

# The two default values below should result in dwz taking at most
# 3GB of RAM or so on 64-bit hosts and 2.5GB on 32-bit hosts
# on the largest *.debug files  (in mid 2012 those are
# libreoffice-debuginfo, debuginfos containing
# libxul.so.debug and libwebkitgtk-*.so.*.debug).
# This needs to be tuned based on the amount of available RAM
# on build boxes for each architecture as well as virtual address
# space limitations if dwz is 32-bit program.  While it needs less
# memory than 64-bit program because pointers are smaller, it can
# never have more than 4GB-epsilon of RAM and on some architecture
# even less than that (e.g. 2GB).

# Number of debugging information entries (DIEs) above which
# dwz will stop considering file for multifile optimizations
# and enter a low memory mode, in which it will optimize
# in about half the memory needed otherwise.
%_dwz_low_mem_die_limit		 10000000
# Number of DIEs above which dwz will stop processing
# a file altogether.
%_dwz_max_die_limit     	 50000000

# On x86_64 increase the higher limit to make libwebkit* optimizable.
# libwebkit* in mid 2012 contains roughly 87mil DIEs, and 64-bit
# dwz is able to optimize it from ~1.1GB to ~410MB using 5.2GB of RAM.
%_dwz_max_die_limit_x86_64	110000000

# On ARM, build boxes often have only 512MB of RAM and are very slow.
# Lower both the limits.
%_dwz_low_mem_die_limit_armv5tel  4000000
%_dwz_low_mem_die_limit_armv7hl	  4000000
%_dwz_max_die_limit_armv5tel	 10000000
%_dwz_max_die_limit_armv7hl	 10000000

%_find_debuginfo_dwz_opts --run-dwz \
  --dwz-low-mem-die-limit %{?_dwz_low_mem_die_limit_%{_arch}} \
  %{!?_dwz_low_mem_die_limit_%{_arch}:%{_dwz_low_mem_die_limit}} \
  --dwz-max-die-limit %{?_dwz_max_die_limit_%{_arch}} \
  %{!?_dwz_max_die_limit_%{_arch}:%{_dwz_max_die_limit}}
=======================
In addition to that, redhat-rpm-config should Requires: dwz and the macros change in #c0 is desirable too.

Can we please move on with this? It is quite urgent.  Thanks.
Comment 5 Panu Matilainen 2012-06-25 09:57:29 UTC 
Sorry, Friday was a holiday for me but lets try to get this sorted out soon. The rpm-side is now in rawhide as of rpm-4.10.0-2.fc18, now on to the redhat-rpm-config part...
Comment 6 Jakub Jelinek 2012-06-25 11:13:53 UTC 
Created attachment 594159 [details]
redhat-rpm-config-9.1.0-dwz.patch

So, this ought to work (macro magic from Panu, thanks).

--- redhat-rpm-config.spec.jj	2012-06-06 14:03:37.000000000 +0200
+++ redhat-rpm-config.spec	2012-06-25 13:10:40.208842781 +0200
@@ -29,10 +29,12 @@ Patch5: redhat-rpm-config-9.1.0-arm.patc
 Patch6: redhat-rpm-config-9.1.0-relro.patch
 Patch7: redhat-rpm-config-9.1.0-hardened.patch
 Patch8: redhat-rpm-config-9.1.0-ppc-no-minimal-toc.patch
+Patch9: redhat-rpm-config-9.1.0-dwz.patch
 BuildArch: noarch
 Requires: coreutils
 Requires: perl-srpm-macros
 Requires: rpm >= 4.6.0
+Requires: dwz >= 0.4
 BuildRequires: libtool
 
 %description
@@ -49,6 +51,7 @@ Red Hat specific rpm configuration files
 %patch6 -p1
 %patch7 -p1
 %patch8 -p1
+%patch9 -p1
 
 %build
Comment 7 Panu Matilainen 2012-06-25 11:54:03 UTC 
Thanks for the updated patch, built into redhat-rpm-config-9.1.0-30.fc18 now.
Comment 8 Panu Matilainen 2012-08-02 06:55:36 UTC 
This appears to be going without complications, time to close the bug...