Bug 1456261 - debuginfo files for Gtk2 plugin process are very large
Summary: debuginfo files for Gtk2 plugin process are very large
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: webkitgtk4
Version: 27
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Tomas Popela
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: 1470692
TreeView+ depends on / blocked
 
Reported: 2017-05-28 13:37 UTC by Christian Stadelmann
Modified: 2019-07-02 12:47 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-11-06 05:21:46 UTC
Type: Bug


Attachments (Terms of Use)

Description Christian Stadelmann 2017-05-28 13:37:41 UTC
Description of problem:
When installing the webkitgtk4 debuginfo package, it is 1.5 GB to download. The /usr/lib/debug/usr/libexec/webkit2gtk-4.0/WebKitPluginProcess2.debug file alone is 2.3 GB in size. This seems wrong, can it be smaller?

Debugging webkitgtk4 often does not require debug symbols for the Gtk2 plugin process. Is it possible to put the Gtk2 plugin debug symbols into a separate package that only gets installed if webkitgtk4-plugin-process-gtk2 is installed?

Version-Release number of selected component (if applicable):
2.16.3-1.fc26

How reproducible:
always – packaging

Steps to Reproduce:
1. run `dnf debuginfo-install webkitgtk4` or `dnf debuginfo-install epiphany`
2. have a look at the file sizes

Comment 1 Michael Catanzaro 2017-05-29 21:40:14 UTC
(In reply to Christian Stadelmann from comment #0)
> Description of problem:
> When installing the webkitgtk4 debuginfo package, it is 1.5 GB to download.
> The /usr/lib/debug/usr/libexec/webkit2gtk-4.0/WebKitPluginProcess2.debug
> file alone is 2.3 GB in size. This seems wrong, can it be smaller?

I think the GTK+ 2 plugin process has its own static-linked copy of WebCore. We *could* cut down on file size by installing a private shared library for the files that don't need GTK+, but that's something that would have to be discussed upstream.

> Debugging webkitgtk4 often does not require debug symbols for the Gtk2
> plugin process. Is it possible to put the Gtk2 plugin debug symbols into a
> separate package that only gets installed if webkitgtk4-plugin-process-gtk2
> is installed?

Hm, I'm surprised that's not already the case....

Comment 2 Yanko Kaneti 2017-06-15 07:40:42 UTC
8 of the The current .debug files for webkitgtk4 (2.17.3) exceed the _dwz_max_die_limit_x86_64 of 110000000 (current redhat-rpm-config) 

Tried locally setting the limit to 200000000, which allowed dwz to halve the size of all those while keeping memory utilization of the dwz to 8-10G in the process.

Maybe we can try
%global _dwz_max_die_limit_x86_64 200000000  
and see how that goes in koji

Comment 3 Tomas Popela 2017-06-15 13:42:31 UTC
(In reply to Yanko Kaneti from comment #2)
> 8 of the The current .debug files for webkitgtk4 (2.17.3) exceed the
> _dwz_max_die_limit_x86_64 of 110000000 (current redhat-rpm-config) 
> 
> Tried locally setting the limit to 200000000, which allowed dwz to halve the
> size of all those while keeping memory utilization of the dwz to 8-10G in
> the process.
> 
> Maybe we can try
> %global _dwz_max_die_limit_x86_64 200000000  
> and see how that goes in koji

Thank you for investigation! I tried to make a 2.17.3 scratch build[0] with your suggestions and the debuginfo size decreased from 5 GB (if comparing with original build[1]) to 2.9 GB and that's a significant decrease.

[0] - https://koji.fedoraproject.org/koji/taskinfo?taskID=20027544
[1] - https://koji.fedoraproject.org/koji/buildinfo?buildID=895797

Comment 4 Yanko Kaneti 2017-06-15 16:34:10 UTC
2.9 (for the rpm) still seems a bit too much. Not long ago the installed size of the debuginfo was around 1.5G if I recall correctly.

from the scratch build log:
dwz: ./usr/lib64/libwebkit2gtk-4.0.so.37.22.0-2.17.3-2.fc27.x86_64.debug: Too many DIEs, not optimizing

Not sure why I didn't see that here. I am guessing we could shave a 1G(installed) there by trying 250000000

Comment 5 Tomas Popela 2017-06-16 08:50:18 UTC
(In reply to Yanko Kaneti from comment #4)
> 2.9 (for the rpm) still seems a bit too much. Not long ago the installed
> size of the debuginfo was around 1.5G if I recall correctly.
> 
> from the scratch build log:
> dwz: ./usr/lib64/libwebkit2gtk-4.0.so.37.22.0-2.17.3-2.fc27.x86_64.debug:
> Too many DIEs, not optimizing
> 
> Not sure why I didn't see that here. I am guessing we could shave a
> 1G(installed) there by trying 250000000

1.1 with 25000000 set - https://koji.fedoraproject.org/koji/taskinfo?taskID=20041364

I will push it to rawhide first to see how it behaves and then to other branches as well.

Comment 6 Tomas Popela 2017-06-16 18:31:50 UTC
I noticed that aarch64 debuginfo package are affected as well. I tried to increase the limit there as well[0], but there were some errors:

dwz: dwz.c:9253: recompute_abbrevs: Assertion `off == cu_size' failed.
/usr/lib/rpm/find-debuginfo.sh: line 418: 25251 Aborted                 (core dumped) dwz $dwz_opts $dwz_files
/usr/lib/rpm/sepdebugcrcfix: Updated 2 CRC32s, 8 CRC32s did match.
cpio: aarch64-redhat-linux-gnu/Source/WebCore/CSSPropertyNames.gperf: Cannot stat: No such file or directory
cpio: aarch64-redhat-linux-gnu/Source/WebCore/CSSValueKeywords.gperf: Cannot stat: No such file or directory
cpio: aarch64-redhat-linux-gnu/Source/WebCore/HTTPHeaderNames.gperf: Cannot stat: No such file or directory
cpio: aarch64-redhat-linux-gnu/Source/WebCore/SelectorPseudoClassAndCompatibilityElementMap.gperf: Cannot stat: No such file or directory
cpio: aarch64-redhat-linux-gnu/Source/WebCore/SelectorPseudoElementTypeMap.gperf: Cannot stat: No such file or directory
cpio: aarch64-redhat-linux-gnu/Source/WebCore/Tokenizer.l: Cannot stat: No such file or directory
cpio: aarch64-redhat-linux-gnu/Source/WebCore/WebCore/xml/XPathGrammar.y: Cannot stat: No such file or directory
cpio: aarch64-redhat-linux-gnu/Source/WebCore/XPathGrammar.cpp: Cannot stat: No such file or directory
cpio: aarch64-redhat-linux-gnu/Source/WebCore/XPathGrammar.hpp: Cannot stat: No such file or directory
cpio: aarch64-redhat-linux-gnu/Source/WebCore/glslang_lex.cpp: Cannot stat: No such file or directory

Maybe it could be caused by not having enough memory in builders? Any idea Peter? It decreased there from ~5.6G to ~4.6G.

[0] - https://koji.fedoraproject.org/koji/taskinfo?taskID=20042012

Comment 7 Peter Robinson 2017-06-18 11:27:19 UTC
> Maybe it could be caused by not having enough memory in builders? Any idea
> Peter? It decreased there from ~5.6G to ~4.6G.

Unlikely, both the ARMv7 and aarch64 builders have 24Gb of RAM (all x86 VM builders have 10)

Comment 8 Patrick Uiterwijk 2017-06-18 23:10:33 UTC
Note that https://kojipkgs.fedoraproject.org/packages/webkitgtk4/2.17.3/1.fc27/aarch64 has a 5.1GB debuginfo rpm, and the respective x86_64 build as well.

Please note that this RPM will be unable to go out soon, since our signing infrastructure cannot handle >4GB at this moment (two places use 32-bit integers to represent size, one of which is koji: https://koji.fedoraproject.org/koji/rpminfo?rpmID=9792187).
While I will be fixing at least the signing infra part, you might want to look at getting new builds for rawhide, since the current ones won't be able to go out soon.

Comment 9 Patrick Uiterwijk 2017-06-18 23:20:07 UTC
The signing software limitation is recorded as bug #1462565.

Comment 10 Tomas Popela 2017-06-19 09:35:21 UTC
(In reply to Patrick Uiterwijk from comment #8)
> While I will be fixing at least the signing infra part, you might want to
> look at getting new builds for rawhide, since the current ones won't be able
> to go out soon.

Building right now as webkitgtk4-2.17.3-2.fc27

Comment 11 Christian Stadelmann 2017-06-20 12:10:21 UTC
(In reply to Michael Catanzaro from comment #1)
> (In reply to Christian Stadelmann from comment #0)
> > Debugging webkitgtk4 often does not require debug symbols for the Gtk2
> > plugin process. Is it possible to put the Gtk2 plugin debug symbols into a
> > separate package that only gets installed if webkitgtk4-plugin-process-gtk2
> > is installed?
> 
> Hm, I'm surprised that's not already the case....

For having separate debuginfo files for subpackage, see this change which has just been announced: https://fedoraproject.org/wiki/Changes/SubpackageAndSourceDebuginfo

I know that the change does not fix the core issue.


Interestingly, on i686, the -debuginfo package is just 70MB: https://kojipkgs.fedoraproject.org//packages/webkitgtk4/2.16.3/1.fc26/i686/

Comment 12 Tomas Popela 2017-06-20 12:16:28 UTC
(In reply to Christian Stadelmann from comment #11)
> Interestingly, on i686, the -debuginfo package is just 70MB:
> https://kojipkgs.fedoraproject.org//packages/webkitgtk4/2.16.3/1.fc26/i686/

From the SPEC file:

# Decrease debuginfo even on ix86 because of:
# https://bugs.webkit.org/show_bug.cgi?id=140176

https://src.fedoraproject.org/cgit/rpms/webkitgtk4.git/tree/webkitgtk4.spec#n158

Comment 13 Patrick Uiterwijk 2017-07-17 10:07:46 UTC
Please note that in the webkitgtk4-2.17.4-1.fc27 build, the x86_64 debuginfo got too big: https://koji.fedoraproject.org/koji/buildinfo?buildID=909563 (https://koji.fedoraproject.org/koji/rpminfo?rpmID=10003418).

Comment 14 Michael Catanzaro 2017-07-17 18:46:26 UTC
Please rebuild epiphany in F27 when this is fixed. It's failing now since it requires 2.17.3.

Comment 15 Tomas Popela 2017-07-19 10:12:04 UTC
(In reply to Michael Catanzaro from comment #14)
> Please rebuild epiphany in F27 when this is fixed. It's failing now since it
> requires 2.17.3.

https://koji.fedoraproject.org/koji/taskinfo?taskID=20609183

Comment 16 Tomas Popela 2017-07-19 10:37:49 UTC
(In reply to Tomas Popela from comment #15)
> (In reply to Michael Catanzaro from comment #14)
> > Please rebuild epiphany in F27 when this is fixed. It's failing now since it
> > requires 2.17.3.
> 
> https://koji.fedoraproject.org/koji/taskinfo?taskID=20609183

In the end it's https://koji.fedoraproject.org/koji/taskinfo?taskID=20609409 I didn't wait for the webkitgtk4 build to hit the repo

Comment 17 Tomas Popela 2017-08-07 04:44:33 UTC
Hmm I see that in the last webkitgtk4 build the size again regressed - for x86_64 the webkitgtk4-debuginfo is ~3.2 GB..

Maybe this could be the cause?:

dwz: dwz.c:9253: recompute_abbrevs: Assertion `off == cu_size' failed.
/usr/lib/rpm/find-debuginfo.sh: line 518:  5059 Aborted                 (core dumped) dwz $dwz_opts ${dwz_files[@]}
/usr/lib/rpm/sepdebugcrcfix: Updated 4 CRC32s, 7 CRC32s did match.

taken from ( https://koji.fedoraproject.org/koji/taskinfo?taskID=21062418 )

Comment 18 Tomas Popela 2017-08-07 04:51:07 UTC
Mark, any idea on comment 17?

Comment 19 Mark Wielaard 2017-08-07 10:45:27 UTC
(In reply to Tomas Popela from comment #18)
> Mark, any idea on comment 17?

(Comment 17):
> Hmm I see that in the last webkitgtk4 build the size again regressed - for
> x86_64 the webkitgtk4-debuginfo is ~3.2 GB..
>
> Maybe this could be the cause?:
> 
> dwz: dwz.c:9253: recompute_abbrevs: Assertion `off == cu_size' failed.
> /usr/lib/rpm/find-debuginfo.sh: line 518:  5059 Aborted                 (core
> dumped) dwz $dwz_opts ${dwz_files[@]}
> /usr/lib/rpm/sepdebugcrcfix: Updated 4 CRC32s, 7 CRC32s did match.
>
> taken from ( https://koji.fedoraproject.org/koji/taskinfo?taskID=21062418 )

Yes, that would certainly explain why the debuginfo would be bigger. The dwz (DWARF compressor) crashed, so no duplication reduction was done.

Why it crashed I don't immediately know. This is an assertion on an invariant that should hold in:

/* Recompute abbrevs for CU.  If any children were collapsed during
   compute_abbrevs, their ->u.p2.die_new_abbrev and ->u.p2.die_new_offset
   fields are no longer available and need to be computed again.  */
static void
recompute_abbrevs (dw_cu_ref cu, unsigned int cu_size)

The assert would trigger if the calculated CU offset at the end doesn't match the given cu_size. I cannot immediately see why/when that would be the case. Maybe Jakub (CCed) immediately recognizes when this could trigger.

Probably best to open a bug against dwz.

Comment 20 Jan Kurik 2017-08-15 06:54:08 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 27 development cycle.
Changing version to '27'.

Comment 21 Christian Stadelmann 2017-11-05 14:02:19 UTC
The size is now down to 887MB (single debuginfo on F26) and 316MB (-plugin-process-gtk2-debuginfo on F27), less than 1GB for all -debuginfo subpackages on F27 accumulated. This looks way better, thank you!

(In reply to Mark Wielaard from comment #19)
> Probably best to open a bug against dwz.

Did someone report a bug against dwz and did it get fixed? If yes, can this bug be closed as fixed?

Comment 22 Tomas Popela 2017-11-06 05:21:46 UTC
> (In reply to Mark Wielaard from comment #19)
> > Probably best to open a bug against dwz.
> 
> Did someone report a bug against dwz and did it get fixed? If yes, can this
> bug be closed as fixed?

No I didn't filled anything. But I checked the some of the latest webkitgtk4 builds and the dwz didn't crashed there.

Comment 23 Tom de Vries 2019-07-02 12:47:06 UTC
(In reply to Christian Stadelmann from comment #21)
> The size is now down to 887MB (single debuginfo on F26) and 316MB
> (-plugin-process-gtk2-debuginfo on F27), less than 1GB for all -debuginfo
> subpackages on F27 accumulated. This looks way better, thank you!
> 
> (In reply to Mark Wielaard from comment #19)
> > Probably best to open a bug against dwz.
> 
> Did someone report a bug against dwz and did it get fixed?

It would be great if somebody could file a dwz report upstream ( https://sourceware.org/bugzilla/enter_bug.cgi?product=dwz ).

Either it is fixed, which would be good to know, or it still needs fixing.

Thanks,
- Tom


Note You need to log in before you can comment on or make changes to this bug.