Created attachment 592863 [details] find-debuginfo.sh patch The https://fedoraproject.org/wiki/Features/DwarfCompressor feature has been approved yesterday. The dwz utility is already included in rawhide, binutils minimal support for readelf and objdump is building right now, gdb support is written and will hopefully be soon included. But we need changes on the rpm and redhat-rpm-config side to enable this. I'd like to see something like the following patch to find-debuginfo.sh (lightly tested so far), which will automatically run the dwz utility on the *.debug files creates by stripping to file earlier in the script. In addition to that I'd appreciate if -g3 could be enabled by default in redhat-rpm-config (something like): --- macros.jj 2010-02-02 10:24:04.000000000 +0100 +++ macros 2012-06-19 09:59:45.647230617 +0200 @@ -177,7 +177,7 @@ unset DISPLAY\ %__find_provides /usr/lib/rpm/redhat/find-provides %__find_requires /usr/lib/rpm/redhat/find-requires -%__global_cflags -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 +%__global_cflags -O2 -g3 -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 # Use these macros to differentiate between RH and other KMP implementation(s). redhat_kernel_module_package 1 and some way how to pass smaller or larger than default values of --dwz-low-mem-die-limit 10000000 and --dwz-max-die-limit 50000000 parameters to find-debuginfo.sh. The numbers depend on the minimum amount of RAM in tbe buildboxes and possibly CPU speed as well, for most of the packages the limits aren't reached, but there are problematic packages with huge amounts of debuginfo - libreoffice has ~ 2.5GB of debuginfo, but nothing really huge, on x86_64 fits into ~ 2GB of RAM or so and on i?86 even less, but then there is libxul.so.debug and libwebkit*so*debug (in several packages ~ .5GB file, in several packages ~ 1.2GB file) on which with huge both limits it could need up to 10GB of RAM when not over low mem die limit and 5GB of RAM when over low mem limit, but below max die limit. webkit has ~ 125million DIEs or so if I remember well. On 32-bit hosts we have the problem that there is some small hard virtual address space limit. The default limits 10mil and 50mil should be reasonable for i386, on x86_64 if buildboxes have say 4GB or better 6GB of RAM available, --dwz-max-die-limit could be increased say up to 150mil. On ARM both the limits should be definitely lowered, both because the VA limit is 2GB, boxes have .5GB of RAM? and because it is terribly slow. Any suggestions how to define per-architecture limits and pass them down into _find_debuginfo_opts ?
Created attachment 592888 [details] find-debuginfo.sh patch Updated find-debuginfo.sh patch that only enables running dwz if --run-dwz option is given.
Created attachment 592903 [details] rpm-dwz.patch Yet another update of the patch, this time to create /usr/lib/debug/.build-id/XX/YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY.debug symlinks to ../../.dwz/%{nvra}.debug files.
Created attachment 593658 [details] rpm-dwz.patch As discussed on IRC, this patch just adds %{?_find_debuginfo_dwz_opts} to find-debuginfo.sh arguments, so that redhat-rpm-config can enable it.
I've tried to come up with /etc/rpm/macros.dwz for redhat-rpm-config, but haven't succeeded so far, my rpm macro skills aren't sufficient: ======================= # Macros for reducing debug info size using dwz(1) utility. # The two default values below should result in dwz taking at most # 3GB of RAM or so on 64-bit hosts and 2.5GB on 32-bit hosts # on the largest *.debug files (in mid 2012 those are # libreoffice-debuginfo, debuginfos containing # libxul.so.debug and libwebkitgtk-*.so.*.debug). # This needs to be tuned based on the amount of available RAM # on build boxes for each architecture as well as virtual address # space limitations if dwz is 32-bit program. While it needs less # memory than 64-bit program because pointers are smaller, it can # never have more than 4GB-epsilon of RAM and on some architecture # even less than that (e.g. 2GB). # Number of debugging information entries (DIEs) above which # dwz will stop considering file for multifile optimizations # and enter a low memory mode, in which it will optimize # in about half the memory needed otherwise. %_dwz_low_mem_die_limit 10000000 # Number of DIEs above which dwz will stop processing # a file altogether. %_dwz_max_die_limit 50000000 # On x86_64 increase the higher limit to make libwebkit* optimizable. # libwebkit* in mid 2012 contains roughly 87mil DIEs, and 64-bit # dwz is able to optimize it from ~1.1GB to ~410MB using 5.2GB of RAM. %_dwz_max_die_limit_x86_64 110000000 # On ARM, build boxes often have only 512MB of RAM and are very slow. # Lower both the limits. %_dwz_low_mem_die_limit_armv5tel 4000000 %_dwz_low_mem_die_limit_armv7hl 4000000 %_dwz_max_die_limit_armv5tel 10000000 %_dwz_max_die_limit_armv7hl 10000000 %_find_debuginfo_dwz_opts --run-dwz \ --dwz-low-mem-die-limit %{?_dwz_low_mem_die_limit_%{_arch}} \ %{!?_dwz_low_mem_die_limit_%{_arch}:%{_dwz_low_mem_die_limit}} \ --dwz-max-die-limit %{?_dwz_max_die_limit_%{_arch}} \ %{!?_dwz_max_die_limit_%{_arch}:%{_dwz_max_die_limit}} ======================= In addition to that, redhat-rpm-config should Requires: dwz and the macros change in #c0 is desirable too. Can we please move on with this? It is quite urgent. Thanks.
Sorry, Friday was a holiday for me but lets try to get this sorted out soon. The rpm-side is now in rawhide as of rpm-4.10.0-2.fc18, now on to the redhat-rpm-config part...
Created attachment 594159 [details] redhat-rpm-config-9.1.0-dwz.patch So, this ought to work (macro magic from Panu, thanks). --- redhat-rpm-config.spec.jj 2012-06-06 14:03:37.000000000 +0200 +++ redhat-rpm-config.spec 2012-06-25 13:10:40.208842781 +0200 @@ -29,10 +29,12 @@ Patch5: redhat-rpm-config-9.1.0-arm.patc Patch6: redhat-rpm-config-9.1.0-relro.patch Patch7: redhat-rpm-config-9.1.0-hardened.patch Patch8: redhat-rpm-config-9.1.0-ppc-no-minimal-toc.patch +Patch9: redhat-rpm-config-9.1.0-dwz.patch BuildArch: noarch Requires: coreutils Requires: perl-srpm-macros Requires: rpm >= 4.6.0 +Requires: dwz >= 0.4 BuildRequires: libtool %description @@ -49,6 +51,7 @@ Red Hat specific rpm configuration files %patch6 -p1 %patch7 -p1 %patch8 -p1 +%patch9 -p1 %build
Thanks for the updated patch, built into redhat-rpm-config-9.1.0-30.fc18 now.
This appears to be going without complications, time to close the bug...