A bit more about what /usr/lib/rpm/find-debuginfo.sh does:


+ /usr/lib/rpm/find-debuginfo.sh --strict-build-id -m --ver-rel 2017.03-1.fc27 --unique-debug-arch aarch64 --unique-debug-src-base uboot-tools --run-dwz --dwz-low-mem-die-limit 10000000 --dwz-max-die-limit 50000000 /home/tkloczko/rpmbuild/BUILD/u-boot-2017.03
extracting debug info from /home/tkloczko/rpmbuild/BUILDROOT/uboot-tools-2017.03-1.fc27.aarch64/usr/bin/bmp_logo
readelf: /home/tkloczko/rpmbuild/BUILDROOT/uboot-tools-2017.03-1.fc27.aarch64/usr/lib/debug/usr/bin/bmp_logo-2017.03-1.fc27.aarch64.debug: Error: the dynamic segment offset + size exceeds the size of the file
extracting debug info from /home/tkloczko/rpmbuild/BUILDROOT/uboot-tools-2017.03-1.fc27.aarch64/usr/bin/fit_check_sign
extracting debug info from /home/tkloczko/rpmbuild/BUILDROOT/uboot-tools-2017.03-1.fc27.aarch64/usr/bin/img2srec
readelf: /home/tkloczko/rpmbuild/BUILDROOT/uboot-tools-2017.03-1.fc27.aarch64/usr/lib/debug/usr/bin/img2srec-2017.03-1.fc27.aarch64.debug: Error: the dynamic segment offset + size exceeds the size of the file
extracting debug info from /home/tkloczko/rpmbuild/BUILDROOT/uboot-tools-2017.03-1.fc27.aarch64/usr/bin/xway-swap-bytes
readelf: /home/tkloczko/rpmbuild/BUILDROOT/uboot-tools-2017.03-1.fc27.aarch64/usr/lib/debug/usr/bin/xway-swap-bytes-2017.03-1.fc27.aarch64.debug: Error: the dynamic segment offset + size exceeds the size of the file
extracting debug info from /home/tkloczko/rpmbuild/BUILDROOT/uboot-tools-2017.03-1.fc27.aarch64/usr/bin/ncb
readelf: /home/tkloczko/rpmbuild/BUILDROOT/uboot-tools-2017.03-1.fc27.aarch64/usr/lib/debug/usr/bin/ncb-2017.03-1.fc27.aarch64.debug: Error: the dynamic segment offset + size exceeds the size of the file
extracting debug info from /home/tkloczko/rpmbuild/BUILDROOT/uboot-tools-2017.03-1.fc27.aarch64/usr/bin/proftool
readelf: /home/tkloczko/rpmbuild/BUILDROOT/uboot-tools-2017.03-1.fc27.aarch64/usr/lib/debug/usr/bin/proftool-2017.03-1.fc27.aarch64.debug: Error: the dynamic segment offset + size exceeds the size of the file
extracting debug info from /home/tkloczko/rpmbuild/BUILDROOT/uboot-tools-2017.03-1.fc27.aarch64/usr/bin/mkimage
extracting debug info from /home/tkloczko/rpmbuild/BUILDROOT/uboot-tools-2017.03-1.fc27.aarch64/usr/bin/gen_eth_addr
readelf: /home/tkloczko/rpmbuild/BUILDROOT/uboot-tools-2017.03-1.fc27.aarch64/usr/lib/debug/usr/bin/gen_eth_addr-2017.03-1.fc27.aarch64.debug: Error: the dynamic segment offset + size exceeds the size of the file
extracting debug info from /home/tkloczko/rpmbuild/BUILDROOT/uboot-tools-2017.03-1.fc27.aarch64/usr/bin/gdbsend
readelf: /home/tkloczko/rpmbuild/BUILDROOT/uboot-tools-2017.03-1.fc27.aarch64/usr/lib/debug/usr/bin/gdbsend-2017.03-1.fc27.aarch64.debug: Error: the dynamic segment offset + size exceeds the size of the file
extracting debug info from /home/tkloczko/rpmbuild/BUILDROOT/uboot-tools-2017.03-1.fc27.aarch64/usr/bin/mkenvimage
readelf: /home/tkloczko/rpmbuild/BUILDROOT/uboot-tools-2017.03-1.fc27.aarch64/usr/lib/debug/usr/bin/mkenvimage-2017.03-1.fc27.aarch64.debug: Error: the dynamic segment offset + size exceeds the size of the file
extracting debug info from /home/tkloczko/rpmbuild/BUILDROOT/uboot-tools-2017.03-1.fc27.aarch64/usr/bin/easylogo
readelf: /home/tkloczko/rpmbuild/BUILDROOT/uboot-tools-2017.03-1.fc27.aarch64/usr/lib/debug/usr/bin/easylogo-2017.03-1.fc27.aarch64.debug: Error: the dynamic segment offset + size exceeds the size of the file
extracting debug info from /home/tkloczko/rpmbuild/BUILDROOT/uboot-tools-2017.03-1.fc27.aarch64/usr/bin/fit_info
extracting debug info from /home/tkloczko/rpmbuild/BUILDROOT/uboot-tools-2017.03-1.fc27.aarch64/usr/bin/gdbcont
readelf: /home/tkloczko/rpmbuild/BUILDROOT/uboot-tools-2017.03-1.fc27.aarch64/usr/lib/debug/usr/bin/gdbcont-2017.03-1.fc27.aarch64.debug: Error: the dynamic segment offset + size exceeds the size of the file
extracting debug info from /home/tkloczko/rpmbuild/BUILDROOT/uboot-tools-2017.03-1.fc27.aarch64/usr/bin/dumpimage
extracting debug info from /home/tkloczko/rpmbuild/BUILDROOT/uboot-tools-2017.03-1.fc27.aarch64/usr/bin/ubsha1
readelf: /home/tkloczko/rpmbuild/BUILDROOT/uboot-tools-2017.03-1.fc27.aarch64/usr/lib/debug/usr/bin/ubsha1-2017.03-1.fc27.aarch64.debug: Error: the dynamic segment offset + size exceeds the size of the file
/usr/lib/rpm/sepdebugcrcfix: Updated 15 CRC32s, 0 CRC32s did match.
1594 blocks

Thanks for the report. This is similar to what is happening to a build of qemu. It contains a non-executable ELF file in a binary package. New rpm code tries to find a build-id in such a file, while the old code didn't.

Testing the following patch now:

diff --git a/build/files.c b/build/files.c
index 6021643..afa01cd 100644
--- a/build/files.c
+++ b/build/files.c
@@ -1711,6 +1711,19 @@ static int generateBuildIDs(FileList fl)
     for (i = 0, flp = fl->files.recs; i < fl->files.used; i++, flp++) {
        struct stat sbuf;
        if (lstat(flp->diskPath, &sbuf) == 0 && S_ISREG (sbuf.st_mode)) {
+           /* We determine whether this is a main or
+              debug ELF based on path.  */
+           #define DEBUGPATH "/usr/lib/debug/"
+           int isDbg = strncmp (flp->cpioPath,
+                                DEBUGPATH, strlen (DEBUGPATH)) == 0;
+
+           /* For the main package files mimic what find-debuginfo.sh does.
+              Only check build-ids for executable files. Debug files are
+              always non-executable. */
+           if (!isDbg
+               && (sbuf.st_mode & (S_IXUSR | S_IXGRP | S_IXOTH)) == 0)
+             continue;
+
            int fd = open (flp->diskPath, O_RDONLY);
            if (fd >= 0) {
                /* Only real ELF files, that are ET_EXEC, ET_DYN or
@@ -1732,12 +1745,8 @@ static int generateBuildIDs(FileList fl)
                       is 128 bits) and 64 bytes (largest sha3 is 512
                       bits), common is 20 bytes (sha1 is 160 bits). */
                    if (len >= 16 && len <= 64) {
-                       /* We determine whether this is a main or
-                          debug ELF based on path.  */
-                       #define DEBUGPATH "/usr/lib/debug/"
                        int addid = 0;
-                       if (strncmp (flp->cpioPath,
-                                    DEBUGPATH, strlen (DEBUGPATH)) == 0) {
+                       if (isDbg) {
                            needDbg = 1;
                            addid = 1;
                        }

Confusingly the readelf errors in the log seem unrelated. I'll open a separate bug about that against binutils.

A similar issue was reported with qemu which also contains a non-executable ELF file in a binary package as a comment to a different bug.
https://bugzilla.redhat.com/show_bug.cgi?id=1433129#c4
Since that bug has already been closed and this issue is slightly different I am handling that as part of this bug.

Posted the patch upstream:
http://lists.rpm.org/pipermail/rpm-maint/2017-March/005262.html
And put it into the new rpm fedora rawhide package (rpm-4.13.0.1-13.fc27).
A mockbuild of qemu and a scratch build of uboot-tools on aarch64 succeed with that.

(In reply to Mark Wielaard from comment #2)
> Confusingly the readelf errors in the log seem unrelated. I'll open a
> separate bug about that against binutils.

This is now https://bugzilla.redhat.com/show_bug.cgi?id=1434050

Just I found yet another issue related to build-id.

Seems it breaks rpmbuild in some other way.
When rpmbuild is executed only to verify %files it start showing on duplicated %files entries on build-id entries.

Please test it for example on parted by:

$ rpmbuild -bi parted.spec --nocheck

(--nocheck added to speedup this test)
And after this

$ rpmbuild -bl --short-circuit parted.spec


This is what I have after second command:

Checking for unpackaged file(s): /usr/lib/rpm/check-files /home/tkloczko/rpmbuild/BUILDROOT/parted-3.2-25.fc27.x86_64
error: Installed (but unpackaged) file(s) found:
   /usr/lib/.build-id/34/dfbcd8dc3d9854803dd00834a96b28ccd98eb8
   /usr/lib/.build-id/52/2c07e88879c4b17b2d0261e7702bf4d2bea0a0
   /usr/lib/.build-id/75/85dac3b1406a284187295c757f790ea0d11aca
   /usr/lib/.build-id/83/ced6d73243074c274e46b907a466932e629f6b
   /usr/lib/debug/.build-id/34/dfbcd8dc3d9854803dd00834a96b28ccd98eb8
   /usr/lib/debug/.build-id/34/dfbcd8dc3d9854803dd00834a96b28ccd98eb8.debug
   /usr/lib/debug/.build-id/52/2c07e88879c4b17b2d0261e7702bf4d2bea0a0
   /usr/lib/debug/.build-id/52/2c07e88879c4b17b2d0261e7702bf4d2bea0a0.debug
   /usr/lib/debug/.build-id/75/85dac3b1406a284187295c757f790ea0d11aca
   /usr/lib/debug/.build-id/75/85dac3b1406a284187295c757f790ea0d11aca.debug
   /usr/lib/debug/.build-id/83/ced6d73243074c274e46b907a466932e629f6b
   /usr/lib/debug/.build-id/83/ced6d73243074c274e46b907a466932e629f6b.debug


RPM build errors:
    Duplicate build-ids /home/tkloczko/rpmbuild/BUILDROOT/parted-3.2-25.fc27.x86_64/sbin/parted and /home/tkloczko/rpmbuild/BUILDROOT/parted-3.2-25.fc27.x86_64/sbin/parted
    Duplicate build-ids /home/tkloczko/rpmbuild/BUILDROOT/parted-3.2-25.fc27.x86_64/sbin/partprobe and /home/tkloczko/rpmbuild/BUILDROOT/parted-3.2-25.fc27.x86_64/sbin/partprobe
    Duplicate build-ids /home/tkloczko/rpmbuild/BUILDROOT/parted-3.2-25.fc27.x86_64/lib64/libparted.so.2.0.1 and /home/tkloczko/rpmbuild/BUILDROOT/parted-3.2-25.fc27.x86_64/lib64/libparted.so.2.0.1
    Duplicate build-ids /home/tkloczko/rpmbuild/BUILDROOT/parted-3.2-25.fc27.x86_64/lib64/libparted-fs-resize.so.0.0.1 and /home/tkloczko/rpmbuild/BUILDROOT/parted-3.2-25.fc27.x86_64/lib64/libparted-fs-resize.so.0.0.1
    Duplicate build-ids /home/tkloczko/rpmbuild/BUILDROOT/parted-3.2-25.fc27.x86_64/usr/lib/debug/lib64/libparted.so.2.0.1-3.2-25.fc27.x86_64.debug and /home/tkloczko/rpmbuild/BUILDROOT/parted-3.2-25.fc27.x86_64/usr/lib/debug/lib64/libparted.so.2.0.1-3.2-25.fc27.x86_64.debug
    Duplicate build-ids /home/tkloczko/rpmbuild/BUILDROOT/parted-3.2-25.fc27.x86_64/usr/lib/debug/lib64/libparted-fs-resize.so.0.0.1-3.2-25.fc27.x86_64.debug and /home/tkloczko/rpmbuild/BUILDROOT/parted-3.2-25.fc27.x86_64/usr/lib/debug/lib64/libparted-fs-resize.so.0.0.1-3.2-25.fc27.x86_64.debug
    Duplicate build-ids /home/tkloczko/rpmbuild/BUILDROOT/parted-3.2-25.fc27.x86_64/usr/lib/debug/sbin/parted-3.2-25.fc27.x86_64.debug and /home/tkloczko/rpmbuild/BUILDROOT/parted-3.2-25.fc27.x86_64/usr/lib/debug/sbin/parted-3.2-25.fc27.x86_64.debug
    Duplicate build-ids /home/tkloczko/rpmbuild/BUILDROOT/parted-3.2-25.fc27.x86_64/usr/lib/debug/sbin/partprobe-3.2-25.fc27.x86_64.debug and /home/tkloczko/rpmbuild/BUILDROOT/parted-3.2-25.fc27.x86_64/usr/lib/debug/sbin/partprobe-3.2-25.fc27.x86_64.debug
    Installed (but unpackaged) file(s) found:
   /usr/lib/.build-id/34/dfbcd8dc3d9854803dd00834a96b28ccd98eb8
   /usr/lib/.build-id/52/2c07e88879c4b17b2d0261e7702bf4d2bea0a0
   /usr/lib/.build-id/75/85dac3b1406a284187295c757f790ea0d11aca
   /usr/lib/.build-id/83/ced6d73243074c274e46b907a466932e629f6b
   /usr/lib/debug/.build-id/34/dfbcd8dc3d9854803dd00834a96b28ccd98eb8
   /usr/lib/debug/.build-id/34/dfbcd8dc3d9854803dd00834a96b28ccd98eb8.debug
   /usr/lib/debug/.build-id/52/2c07e88879c4b17b2d0261e7702bf4d2bea0a0
   /usr/lib/debug/.build-id/52/2c07e88879c4b17b2d0261e7702bf4d2bea0a0.debug
   /usr/lib/debug/.build-id/75/85dac3b1406a284187295c757f790ea0d11aca
   /usr/lib/debug/.build-id/75/85dac3b1406a284187295c757f790ea0d11aca.debug
   /usr/lib/debug/.build-id/83/ced6d73243074c274e46b907a466932e629f6b
   /usr/lib/debug/.build-id/83/ced6d73243074c274e46b907a466932e629f6b.debug
[tkloczko@domek SPECS]$ 


I'm not sure am I understand how build-id is working but seems it is about finding exact debug packages for gdb and other which should be installed to have full and correct debuginfo.
Already each ELF binary have injected into header build-id

[tkloczko@domek sbin]$ eu-readelf -n parted 

Note section [ 2] '.note.ABI-tag' of 32 bytes at offset 0x254:
  Owner          Data size  Type
  GNU                   16  VERSION
    OS: Linux, ABI: 2.6.32

Note section [ 3] '.note.gnu.build-id' of 36 bytes at offset 0x274:
  Owner          Data size  Type
  GNU                   20  GNU_BUILD_ID
    Build ID: 522c07e88879c4b17b2d0261e7702bf4d2bea0a0


Instead building whole directory tree with those symlinks IMO better would be change core file process to store in core files main executable and all used DSOs build id.
If debuginfo packages will have in Provides: list stored:

Provides: build-id(<hash>)

Instead looking on symlinks in build-id tree then after into rpm database to find which one package name provides some binary it could be possible to install straight necessary debuginfo packages by:

# dnf install "build-id(<hash>)"

and dnf/yum automatically will find which one debuginfo package will need to be installed as long as it will be no hash clashes of course.

Seems all /usr/lib/build-id tree duplicates informations which are already partially stored in rpm database. Now even when someone does not need to have debuginfo has in /usr/lib/build-id all binaries paths which are already registered in rpm database. This is like second such database but organized not in db files in whole directories structures.
Current /usr/lib/build-id tree as well have directories which are owned by multiple packages.

Maybe I'm wrong about how build-id should be or how it is already working working but seems something is fundamentally wrong about how this lookup mechanism should be implemented (despite current issues with clashes).

Correct me if I'm wrong but with storing necessary hashes in core files all this directory structure seems will be not needed.

Just checked and seems really now core files are without build-id hashes ans seems whole /usr/lib/build-id is to solve absence of those informations in core files.

(In reply to Tomasz Kłoczko from comment #6)
> Just I found yet another issue related to build-id.
> 
> Seems it breaks rpmbuild in some other way.
> When rpmbuild is executed only to verify %files it start showing on
> duplicated %files entries on build-id entries.
> 
> Please test it for example on parted by:
> 
> $ rpmbuild -bi parted.spec --nocheck
> 
> (--nocheck added to speedup this test)
> And after this
> 
> $ rpmbuild -bl --short-circuit parted.spec

If you think this should be fixed then could you please open a new bug?
This bug is resolved and close and this is clearly a completely different issue.
It might get forgotten if added here.

I am not sure --short-circuit builds are supported. I have never used them myself. It would be nice to know what the precise semantics are of such a build.   

If the build skips the build phase then it seems obvious why you are seeing issues with --short-circuit. the build-id files are created and added to the file list at the same time. If you don't build the file list again then the files will be there but unadded.

It looks like --short-circuit is only supported for local testing. If this is blocking you then you probably need to skip check-files or build with %_build_id_links defined to none (see macros for an overview of the settings).

> I'm not sure am I understand how build-id is working but seems it is about
> finding exact debug packages for gdb and other which should be installed to
> have full and correct debuginfo.
> Already each ELF binary have injected into header build-id

This is also an interesting discussion, but maybe not directly in this bug report.

build-ids are just unique identifiers for executable ELF files that identify is specific build of the binary. That is why in the latest rpm we try to make sure that different rpm builds always produce a unique ID. One usage is indeed to match a binary with its debuginfo. But there are also other identification services. Like darkserver which can map build-ids to packages:
https://darkserver.fedoraproject.org/

See also https://fedoraproject.org/wiki/Releases/FeatureBuildId and https://fedoraproject.org//wiki/Changes/ParallelInstallableDebuginfo
> Instead building whole directory tree with those symlinks IMO better would
> be change core file process to store in core files main executable and all
> used DSOs build id.

This is already done. If you have a core file you can extract the build-ids with something like: eu-unstrip -n --core core.2218

> If debuginfo packages will have in Provides: list stored:
> 
> Provides: build-id(<hash>)
> 
> Instead looking on symlinks in build-id tree then after into rpm database to
> find which one package name provides some binary it could be possible to
> install straight necessary debuginfo packages by:
> 
> # dnf install "build-id(<hash>)"

Yes, this is a good idea. It has already been implemented (partially) in upstream rpm.
  
> Correct me if I'm wrong but with storing necessary hashes in core files all
> this directory structure seems will be not needed.

Directory lookups are just one mechanism (which works with or without rpm). Adding build-ids as provides to rpm packages would be another (but would miss a direct mapping to the actual ELF file). It would certainly be nice to discuss various ideas around this, but lets do that on the upstream rpm or fedora mailinglists and not in this (now closed) bug.

(In reply to Mark Wielaard from comment #8)
> > $ rpmbuild -bi parted.spec --nocheck
> > 
> > (--nocheck added to speedup this test)
> > And after this
> > 
> > $ rpmbuild -bl --short-circuit parted.spec
> 
> If you think this should be fixed then could you please open a new bug?
> This bug is resolved and close and this is clearly a completely different
> issue.
> It might get forgotten if added here.

Opened https://bugzilla.redhat.com/show_bug.cgi?id=1434235

> I am not sure --short-circuit builds are supported. I have never used them
> myself. It would be nice to know what the precise semantics are of such a
> build.   
> 
> If the build skips the build phase then it seems obvious why you are seeing
> issues with --short-circuit. the build-id files are created and added to the
> file list at the same time. If you don't build the file list again then the
> files will be there but unadded.

--short-circuit starts from exact state of prev state of build package procedure.
If let's say you have log building package and something failed on %files verification after correcting %files you can only fire -bl --short-circuit should only do %files verification. I've not been looking closer but I suppose that now %pre/%post %files verifications are adding some new entries dynamically introducing duplicates. If it is correct guess it may mean that build-id was hooked in wrong stage (lo late) breaking -bl --short-circuit.

It is bunch of rules about how to write specs not breaking --short-circuit on every stage. Seems those rules are not documented in Fedora. I cannot promise but will try to write about this somewhere.

> It looks like --short-circuit is only supported for local testing. If this
> is blocking you then you probably need to skip check-files or build with
> %_build_id_links defined to none (see macros for an overview of the
> settings).

--short-circuit it is not for testing. It is powerful development mechanism allowing save a lot of time on perfecting spec file.
I guess that you never been using --short-circuit because probably yu been using more or less working/finished specs files.

[..]
> > # dnf install "build-id(<hash>)"
> 
> Yes, this is a good idea. It has already been implemented (partially) in
> upstream rpm.

So this approach with /usr/lib/build-id is kind of interim solution and in some point of developing whole infrastructure it will be abandoned?
If yes I have nothing to add ..

Thanks for URLs. Will try to read those doc.

> > Correct me if I'm wrong but with storing necessary hashes in core files all
> > this directory structure seems will be not needed.
> 
> Directory lookups are just one mechanism (which works with or without rpm).
> Adding build-ids as provides to rpm packages would be another (but would
> miss a direct mapping to the actual ELF file). It would certainly be nice to
> discuss various ideas around this, but lets do that on the upstream rpm or
> fedora mailinglists and not in this (now closed) bug.

Hooking such infrastructure on "with or without A-feature" makes such things by definition unstable and/or unpredictable.
As long as Fedora uses package manager it make sense to focus on provide solution basing on this that in system already is packages database.
People are able to do incredibly stupid things and some solutions must relay on some exact context.
I'm asking only to focus on develop such technologies as not something which will solve "Earth famin problems" :)