Created attachment 984937 [details] readelf -S of pre stripped files Description of problem: I ran into this problem while attempting to package https://github.com/facebook/hhvm for Fedora/EPEL. When debuginfo package generation is enabled, it runs eu-strip in /usr/lib/rpm/find-debuginfo.sh which strips important sections from the compiled executables. Once the stripping happens, then the executable errors with: Failed to find/load systemlib.php I have (or will have) attached pre and post eu-strip output. Code where sections are added: https://github.com/facebook/hhvm/blob/cb02e599fab0e24c0f8e90b2cfd64c6e72fb7c6f/CMake/HPHPFunctions.cmake#L100 Upstream Ticket: https://github.com/facebook/hhvm/issues/4724 Version-Release number of selected component (if applicable): elfutils-0.161.2.fc20 How reproducible: Everytime Steps to Reproduce: 1. build use provided spec file. 2. 3. Actual results: see Failed to load/find systemlib.php in build failure in rpm error output Expected results: It does not fail there. Additional info:
Created attachment 984938 [details] readelf -S of post stripped files
i should also mention the actual important parts that ARE getting stripped instead of forcing you to look at the the links. The parts that are getting stripped are : systemlib and all ext.* sections
I think this is expected behaviour. The sections aren't marked as needed during runtime, so eu-strip will remove them and put them in the separate .debug file. If these sections are important and needed at runtime, why aren't they marked with SHF_ALLOC?
> If these sections are important and needed at runtime, why aren't they marked with SHF_ALLOC? I explained this in the upstream ticket: https://github.com/facebook/hhvm/issues/4724#issuecomment-71758826 The sections aren't marked SHF_ALLOC because HHVM doesn't read them directly out of the address space -- they don't need to be loaded into memory when the ELF image is loaded. (And of course doing so would be a waste.) Instead, HHVM opens its own binary and reads the sections back out that way. There's probably an argument that doing it that way is a suboptimal way to do it, but it's the way things currently are set up, and it's perfectly valid. AFAIK we haven't had problems with any other stripping or debug symbol tools eating random ELF sections -- notably I was unable to convince `strip` to remove those sections no matter how aggressively I told it to get rid of extra sections. So this issue appears unique to `eu-strip`. Unless I misunderstand what SHF_ALLOC means.
(In reply to Josh Watzman from comment #4) > > If these sections are important and needed at runtime, why aren't they marked with SHF_ALLOC? > > I explained this in the upstream ticket: > https://github.com/facebook/hhvm/issues/4724#issuecomment-71758826 > > The sections aren't marked SHF_ALLOC because HHVM doesn't read them directly > out of the address space -- they don't need to be loaded into memory when > the ELF image is loaded. (And of course doing so would be a waste.) It wouldn't be a waste, they are only mapped into the address space, not actually loaded till you access them. > Instead, > HHVM opens its own binary and reads the sections back out that way. There's > probably an argument that doing it that way is a suboptimal way to do it, > but it's the way things currently are set up, and it's perfectly valid. It is valid yes. You can open any ELF file and read the section data if it is there. Even for the currently running binary. But I think it would indeed be more efficient to just let those segments be mapped in if you are going to use them anyway. And marking them SHF_ALLOC would make clear to other tools dealing with ELF files those sections are actually needed at runtime. > AFAIK we haven't had problems with any other stripping or debug symbol tools > eating random ELF sections -- notably I was unable to convince `strip` to > remove those sections no matter how aggressively I told it to get rid of > extra sections. So this issue appears unique to `eu-strip`. They are not random :) They are just normal PROG_BITS sections not marked SHF_ALLOC. And eu-strip interprets that as unneeded (debug) data that can be moved away without disrupting the runtime execution of the binary file. Currently there is no option in eu-strip to filter out some sections except by setting the section type & flags. If you really don't want to run find-debuginfo.sh while creating a package you can set %define debug_package %{nil} in the spec file. Of course that just won't create any separate debuginfo files. Which might also not be what you want. Sorry.
It used to be the case that setting mode -x on a file removed it from consideration for debuginfo stripping. I think you then needed to use %attr in %files section to install with +x again. Haven't used this in years, but you might want to check out whether it works around the problem.
I tried added alloc flags (although i don't really know how to properly specify where they are located in memory), but it bloated the end result binary from around 51MB to 269MB) Note that debuginfo package generation seems to work on PLDLinux and OpenSUSE just fine, because they use objcopy and not eu-strip. Fedora is the odd one out here. Of course PLDLinux and OpenSUSE don't offer minidebug builds or dwarf compression either. Is there any reason that eu-strip doesn't act like regular strip and contains the -w and -K options (wildcard and keep-sections) ?
(In reply to Johnny Robeson from comment #7) > I tried added alloc flags (although i don't really know how to properly > specify where they are located in memory), but it bloated the end result > binary from around 51MB to 269MB) The phdrs should indicate where the segments are loaded in the final binary. Apart from some changes in the header flags there should be no size difference. But I admit to not have looked at how you are using the sections. > Note that debuginfo package generation seems to work on PLDLinux and > OpenSUSE just fine, because they use objcopy and not eu-strip. Fedora is the > odd one out here. Of course PLDLinux and OpenSUSE don't offer minidebug > builds or dwarf compression either. I think your use case is uncommon. Nobody ever reported using sections in binary executable files this way or having a problem with moving non-allocated sections into the separate .debug files. Of course it could be that they just never bothered to tell us and just disabled generation of debug_package. I haven't looked. If there is an easy way to grep all spec files to see if they set debug_package to nil (or do Petr's -x/+x trick) that would be interesting. > Is there any reason that eu-strip doesn't act like regular strip and > contains the -w and -K options (wildcard and keep-sections) ? Nobody asked for them before. But note that those work on symbols, not sections. So they wouldn't help in this case. We do have a list of "debug section names" (see default_debug_scn_p), so we could add an option to eu-strip to only consider those and ignore the SHF_ALLOC section flag.
i assume the size difference is probably due to it causing the debug binary not to be as stripped as it normally otherwise would be. For any future folks running into this bug: it's default_debugscn_p (not default_debug_scn_p). Is there a quick way i can test to see if only considering default_debug_scn_p will work? My C knowledge is limited to only being able to write (mostly) and not really write.
that'd be: limited to only being able to read (mostly) and not write.
(In reply to Johnny Robeson from comment #0) > Steps to Reproduce: > 1. build use provided spec file. Sorry, where is the provided spec file?
I am interested in whether just using -g helps by adding something like: %define _find_debuginfo_opts -g
i alrady tried -g, it didn't help.
(In reply to Johnny Robeson from comment #13) > i alrady tried -g, it didn't help. Thanks. Could you show me how? Do you happen to have a spec file handy. I would like to replicate the issue.
Sorry for not adding the spec file, but you'll need more than that anyways. Here is a list of instructions to attempt to reproduce $ git clone https://github.com/no1youknowz/hhvm-repo/ $ cd hhvm-repo $ git checkout re2465a5315b8cbbbbad8ef6e3b13599faf852fda9 $ cd hhvm # edit hhvm.spec to remove %global debug_package %{nil} $ ./repack.sh HHVM-3.6.0 # the repack script seems to drop stuff in a directory below, sorry $ mv ../hhvm-3.6.0.tar.gz and other Source* to wherever you use. $ rpmbuild -ba <or whatever you use> It should fail during the %check step. Hopefully this is clear enough, I've never really tried to describe the process before, and have likely not setup my rpm repos correctly.
oh yeah, forgot to mention to run yum builddep or dnf builddep! PS: sorry for the excessive dependencies in third-party, i haven't yet gotten them all split out into their own packages.
OH.. here's an SRPM! that'd be a lot easier! https://copr-be.cloud.fedoraproject.org/results/jrobeson/hhvm/fedora-21-x86_64/hhvm-3.6.0-3.fc21/hhvm-3.6.0-3.fc21.src.rpm
Sorry, the build keeps failing for me without a clear reason: /usr/bin/ranlib libfolly.a make[2]: Leaving directory '/home/mark/BUILD/hhvm-3.6.0' /usr/bin/cmake -E cmake_progress_report /home/mark/BUILD/hhvm-3.6.0/CMakeFiles 2 3 4 5 6 7 [ 16%] Built target folly make[1]: Leaving directory '/home/mark/BUILD/hhvm-3.6.0' Makefile:120: recipe for target 'all' failed make: *** [all] Error 2 Could you try replacing: --- hhvm.spec.orig 2015-03-17 13:15:02.626443010 +0100 +++ hhvm.spec 2015-03-17 13:16:31.539224229 +0100 @@ -10,8 +10,9 @@ #TODO: provide php alternative https://fedoraproject.org/wiki/Packaging:Alternatives #TODO: find a way to get PEAR to work with hhvm, so we can provide php-common too. %global _hardened_build 1 -#TODO: reenable debug builds https://bugzilla.redhat.com/show_bug.cgi?id=1186563 -%global debug_package %{nil} +# Force strip of only the debug sections since some non-allocated sections +# are used during runtime. +%global _find_debuginfo_opts -g %{!?_httpd_confdir: %{expand: %%global _httpd_confdir %%{_sysconfdir}/httpd/conf.d}} # httpd 2.4.10 with httpd-filesystem and sethandler support %if 0%{?fedora} >= 21 And post the build log? Thanks.
what OS are you trying to build on ? is it on Fedora 20 or 21? If it's 20, then can you try 21? i have tried added -g before, but will try again.
(In reply to Johnny Robeson from comment #19) > what OS are you trying to build on ? is it on Fedora 20 or 21? If it's 20, > then can you try 21? f21 x86_64. > i have tried added -g before, but will try again. Thanks. sorry I didn't get it to work locally. I am interested what the result is because theoretically it should do what you need.
Created attachment 1003093 [details] build.log
I commented out the place it would normally fail in %check, just so you can see that the rpm is built successfully Here is a link to the sucessfully built, but broken RPM: http://hhvm-dev.localmomentum.net/hhvm-3.6.0-3.fc21.x86_64.rpm You can easily reproduce the error like this: $ yum install http://hhvm-dev.localmomentum.net/hhvm-3.6.0-3.fc21.x86_64.rpm $ hhvm --php -r 'exit(0);' You will then see: Failed to load/find systemlib.php Here is the exact SRPM i used to build it from (with mock): http://hhvm-dev.localmomentum.net/hhvm-3.6.0-3.fc21.src.rpm If you'd like to examine the binary before it's stripped (without building), then you can download it here http://hhvm-dev.localmomentum.net/hhvm-prestripped. It should work as long as you have have all the dependencies installed. WARNING: This file is 1.2GB, so be ready for that! You can then reproduce the failure by: $ chmod +x hhvm-prestripped $ cp hhvm-prestripped hhvm # keep a backup copy! $ eu-strip <whatever args you want> $ hhvm --php -r 'exit(0);' If you still have problems building, and this binary isn't enough (somehow), then i could tar up the entire mock chroot and then you can examine everything in the builddir yourself.
Thanks you. I finally see what is going wrong. And it was not where I was looking. The problem isn't in eu-strip. The issue is that find-debuginfo.sh only passes -g for shared libraries, not for executables. The following would make things work as expected with %global _find_debuginfo_opts -g --- /usr/lib/rpm/find-debuginfo.sh 2015-01-12 12:19:57.000000000 +0100 +++ find-debuginfo.sh 2015-03-18 12:45:53.207743271 +0100 @@ -130,6 +130,7 @@ $strip_r && r=--reloc-debug-sections $strip_g && case "$(file -bi "$2")" in application/x-sharedlib*) g=-g ;; + application/x-executable*) g=-g ;; esac eu-strip --remove-comment $r $g -f "$1" "$2" || exit chmod 444 "$1" || exit Lets reassign this issue to rpm (rpmbuild provides the find-debuginfo.sh script) and see if we can figure out why that restriction to shared libraries is there in the first place and if we can work around it somehow.
Roland, I see that you introduced the restriction of passing -g given to find-debuginfo.sh to eu-strip only for shared-libraries. It was 8 years ago that code was added, but maybe you still remember why that was done? Would it be better to just remove that restriction and always pass on -g to eu-strip when requested, or extend it to cover only sharedlib and executables (so it wouldn't impact kernel modules), or add a separate flag -g-exec to pass -g to eu-strip for executables?
Note that for PIE executables it would work since file -bi returns application/x-sharedlib for those.
i manually edited find-debuginfo.sh to force passing -g directly on eu-strip and that gives me this result in the build.log: extracting debug info from /builddir/build/BUILDROOT/hhvm-3.6.0-3.fc21.x86_64/usr/bin/hh_client nm: /builddir/build/BUILDROOT/hhvm-3.6.0-3.fc21.x86_64/usr/lib/debug/usr/bin/hh_client.debug: no symbols xz: /tmp/tmp.GKY2FJumZC: No such file or directory objcopy: cannot open: /tmp/tmp.GKY2FJumZC.xz: No such file or directory extracting debug info from /builddir/build/BUILDROOT/hhvm-3.6.0-3.fc21.x86_64/usr/bin/hh_server nm: /builddir/build/BUILDROOT/hhvm-3.6.0-3.fc21.x86_64/usr/lib/debug/usr/bin/hh_server.debug: no symbols xz: /tmp/tmp.fTpjZC7pZQ: No such file or directory objcopy: cannot open: /tmp/tmp.fTpjZC7pZQ.xz: No such file or directory extracting debug info from /builddir/build/BUILDROOT/hhvm-3.6.0-3.fc21.x86_64/usr/bin/hhvm nm: /builddir/build/BUILDROOT/hhvm-3.6.0-3.fc21.x86_64/usr/lib/debug/usr/bin/hhvm.debug: no symbols xz: /tmp/tmp.b0JTDlV0Qr: No such file or directory objcopy: cannot open: /tmp/tmp.b0JTDlV0Qr.xz: No such file or directory It can't seem to find any debug symbols.
Mark Wielaard: did you try running eu-strip manually on that prestripped binary i linked to? did it actually work? I tried before on a system running Fedora 20 (when i initially filed this bug), and even then it did not result in a working binary.
(In reply to Johnny Robeson from comment #26) > i manually edited find-debuginfo.sh to force passing -g directly on eu-strip > and that gives me this result in the build.log: > > extracting debug info from > /builddir/build/BUILDROOT/hhvm-3.6.0-3.fc21.x86_64/usr/bin/hhvm > nm: > /builddir/build/BUILDROOT/hhvm-3.6.0-3.fc21.x86_64/usr/lib/debug/usr/bin/ > hhvm.debug: no symbols > xz: /tmp/tmp.b0JTDlV0Qr: No such file or directory > objcopy: cannot open: /tmp/tmp.b0JTDlV0Qr.xz: No such file or directory > > > It can't seem to find any debug symbols. That comes from the (misnamed) add_minidebug () in find-debuginfo.sh. Which tries to extract from the .debug file some of the function symbols of the symtab to put in the main executable. With eu-strip -g we only strip the .debug sections, not the symtab (or other none loaded sections). So there is no symtab in the .debug file. It is still in the main executable. So the minisymtab is unnecessary. Is the above a real failure or just warnings? If it produces an error then you might want to include the following in your spec file: # Force strip of only the debug sections since some non-allocated sections # are used during runtime. Since we keep the symtab with -g we also don't # want minisymtab since it is unnecessary (we have the full symtab). %define _find_debuginfo_opts -g %undefine _include_minidebuginfo in your spec file
that's still predicated on getting something like this added to find-debuginfo.sh though. --- /usr/lib/rpm/find-debuginfo.sh 2015-01-12 12:19:57.000000000 +0100 +++ find-debuginfo.sh 2015-03-18 12:45:53.207743271 +0100 @@ -130,6 +130,7 @@ $strip_r && r=--reloc-debug-sections $strip_g && case "$(file -bi "$2")" in application/x-sharedlib*) g=-g ;; + application/x-executable*) g=-g ;; esac eu-strip --remove-comment $r $g -f "$1" "$2" || exit chmod 444 "$1" || exit I'm assuming that this might get patched up for Fedora, but what can i do for RHEL7/CentOS7?
(In reply to Johnny Robeson from comment #27) > Mark Wielaard: did you try running eu-strip manually on that prestripped > binary i linked to? did it actually work? Yes. I ran a tweaked find-debuginfo.sh on it to pass the -g and it produced: -rwxr-xr-x. 1 mark mark 64M Mar 18 12:44 ./usr/bin/hhvm-prestripped -r--r--r--. 1 mark mark 838M Mar 18 12:44 ./usr/lib/debug/usr/bin/hhvm-prestripped.debug And the (now misnamed) ./usr/bin/hhvm-prestripped seems to work: ./usr/bin/hhvm-prestripped --php -r 'print("hello\n"); exit(0);' hello
(In reply to Johnny Robeson from comment #29) > that's still predicated on getting something like this added to > find-debuginfo.sh though. > > --- /usr/lib/rpm/find-debuginfo.sh 2015-01-12 12:19:57.000000000 +0100 > +++ find-debuginfo.sh 2015-03-18 12:45:53.207743271 +0100 > @@ -130,6 +130,7 @@ > $strip_r && r=--reloc-debug-sections > $strip_g && case "$(file -bi "$2")" in > application/x-sharedlib*) g=-g ;; > + application/x-executable*) g=-g ;; > esac > eu-strip --remove-comment $r $g -f "$1" "$2" || exit > chmod 444 "$1" || exit > > I'm assuming that this might get patched up for Fedora I hope so, but that is up to the rpm maintainers. > but what can i do for RHEL7/CentOS7? You can clone the bug and hope they also fix it there. There is probably some way to override debug_package completely and define __debug_package 1 with your own __debug_install_post to use your own script to replace the system find-debuginfo.sh. But better to just get this fixed everywhere (if only because I find overriding such things in the spec file confusing and error prone).
Well, you can always in the spec file just wrap eu-strip with a small shell wrapper that will for some particular arguments (your special binary) tweak the options, then exec the original eu-strip and prepend a directory with the wrapper into your $PATH in the spec file. I believe find-debuginfo.sh doesn't use absolute paths to eu-strip. Or e.g. look at gcc.spec in Fedora 22+ how it deals with eu-unstripping again go binaries which are required for proper running of the binaries (similar misdesign case as in the hhvm case, relying on non-alloc sections for program operation). You can just objcopy the sections back or something similar.
Can you be a bit more specific about how i could do the shell script wrapper? This is my first rpm package, so don't really know eveyrthing about what calls what and when, and when i could even override the path.
As I said, have a look at gcc.spec. Or say: mkdir strip-wrapper/ cat > strip-wrapper/eu-strip <<\EOF #!/bin/sh case " $* " in) */hhvm\ *) exec /usr/bin/eu-strip .... ;; esac exec /usr/bin/eu-strip "$@" EOF chmod 755 strip-wrapper/eu-strip PATH=`pwd`/strip-wrapper/:$PATH (how do you detect when it is called for the binary and how you tweak the option is up to you, just a matter of shell programming) somewhere in the %install section of the spec file (or in %build section, just with the PATH= line in %install section). I believe find-debuginfo.sh is executed at the end of %install snippet, so whatever PATH is set during it will be preserved even for find-debuginfo.sh.
Created attachment 1003845 [details] find-debuginfo.sh: Also pass -g to eu-strip for executables. I submitted a patch upstream to the rpm-maint.org but it seems not yet in the archives. So attached here.
Created attachment 1003848 [details] strip -g implies we have full symtab, don't add mini symtab in that case. The mini symtab support seems to not be merged upstream yet. So here is a patch against the rpm patches as they are in fedora to make sure add_minidebug is not ran when strip_g is set.
thanks Jakub Jelinek! it's working well. I followed his instructions, plus added: %undefine _include_minidebuginfo to turn off minidebug Thanks for your help everyone. Hopefully we will see a patched RPM eventually, but for now I'm good!
I realize that the binary is about double the size it should be. The PLD folks have their hhvm executable as 32MB. But simply adding strip -g (and the rest of the process in find-debuginfo.sh) results in the executable being 69MB, so strip -g isn't going to be enough.
(In reply to Johnny Robeson from comment #38) > I realize that the binary is about double the size it should be. The PLD > folks have their hhvm executable as 32MB. But simply adding strip -g (and > the rest of the process in find-debuginfo.sh) results in the executable > being 69MB, so strip -g isn't going to be enough. What is PLD and what is the difference in the executable are there any specific sections missing/bigger/smaller? This might be a different bug though. If so, please open a new one to track. Lets keep this one for making sure rpm find-debuginfo.sh passes -g correctly to eu-strip.
I didn't change the title to mean that, you did :) They don't even use -g at all. the difference with pld is that they run eu-strip like this: local t=$(mktemp "/tmp/rpm.stripped.XXXXXX") objcopy --compress-debug-sections "$2" || exit eu-strip --remove-comment -f "$1" "$2" -o "$t" || exit rm -f "$t"
oh, but they don't have the dwarf compressor, or minidebug either.
(In reply to Johnny Robeson from comment #40) > I didn't change the title to mean that, you did :) Sure, to better describe the real issue. But really, we are discussion something different now. Lets open a new bug for this if you want to track it. First you got a broken executable since there was no way to strip debuginfo while keeping some non-alloc sections leading to a broken binary. Lets keep this bug open to track that issue. Now we are discussing what seems to be some size optimization issue. > They don't even use -g at all. Who are they? And could you show what their binary and separate debug info file look like (eu-readelf -S). But in a new bug report please if you think it is a bug in fedora.
Upstream patch posted to rpm-maint: http://lists.rpm.org/pipermail/rpm-maint/2015-March/003865.html
Added the first patch to upstream and both patches to rawhide.
(In reply to Mark Wielaard from comment #24) > Roland, I see that you introduced the restriction of passing -g given to > find-debuginfo.sh to eu-strip only for shared-libraries. It was 8 years ago > that code was added, but maybe you still remember why that was done? > > Would it be better to just remove that restriction and always pass on -g to > eu-strip when requested, or extend it to cover only sharedlib and > executables (so it wouldn't impact kernel modules), or add a separate flag > -g-exec to pass -g to eu-strip for executables? I suspect that -g was used for glibc and there it's appropriate to strip-debug all the libraries but fully strip all the executables. The executables are in a separate sub-rpm, but at least at the time everything was together in a single debuginfo rpm and so there was no way to apply different find-debuginfo.sh options to the debuginfo corresponding to the different sub-rpms.