Bug 1986382

Summary: failed symlink(File exists) .build-id error in Processing files with fedora-34-x86_64 mock config
Product: Red Hat Enterprise Linux 8 Reporter: Anoop C S <anoopcs>
Component: rpmAssignee: Packaging Maintenance Team <packaging-team-maint>
Status: CLOSED DUPLICATE QA Contact: swm-qe
Severity: medium Docs Contact:
Priority: unspecified    
Version: 8.4CC: ffesti, mdomonko, packaging-team-maint
Target Milestone: betaFlags: pm-rhel: mirror+
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-08-10 15:31:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
build.log none

Description Anoop C S 2021-07-27 12:13:34 UTC
Created attachment 1806311 [details]
build.log

Description of problem:
On a CentOS 8 Linux machine, build process failed with the following error:

Processing files: python3-samba-20210721.071900.84b9f58616e-1.fc34.1.x86_64
error: failed symlink: /builddir/build/BUILDROOT/samba-20210721.071900.84b9f58616e-1.fc34.1.x86_64/usr/lib/.build-id/09/c2ec9b477081d605387b45fbf3e2de3832d04d -> ../../../../usr/lib64/python3.9/site-packages/samba/dsdb.cpython-39-x86_64-linux-gnu.so: File exists
error: Generating build-id links failed
RPM build errors:
    failed symlink: /builddir/build/BUILDROOT/samba-20210721.071900.84b9f58616e-1.fc34.1.x86_64/usr/lib/.build-id/09/c2ec9b477081d605387b45fbf3e2de3832d04d -> ../../../../usr/lib64/python3.9/site-packages/samba/dsdb.cpython-39-x86_64-linux-gnu.so: File exists
    Generating build-id links failed
Child return code was: 1
EXCEPTION: [Error()]
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.6/site-packages/mockbuild/util.py", line 600, in do_with_status
    raise exception.Error("Command failed: \n # %s\n%s" % (command, output), child.returncode)
mockbuild.exception.Error: Command failed: 
 # /usr/bin/systemd-nspawn -q -M 11caae70857e478593bf03ae754e5ab5 -D /var/lib/mock/fedora-34-x86_64/root -a -u mockbuild --capability=cap_ipc_lock --bind=/tmp/mock-resolv.gknfivk9:/etc/resolv.conf --bind=/dev/loop-control --bind=/dev/loop0 --bind=/dev/loop1 --bind=/dev/loop2 --bind=/dev/loop3 --bind=/dev/loop4 --bind=/dev/loop5 --bind=/dev/loop6 --bind=/dev/loop7 --bind=/dev/loop8 --bind=/dev/loop9 --bind=/dev/loop10 --bind=/dev/loop11 --setenv=TERM=vt100 --setenv=SHELL=/bin/bash --setenv=HOME=/builddir --setenv=HOSTNAME=mock --setenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin --setenv=PROMPT_COMMAND=printf "\033]0;<mock-chroot>\007" --setenv=PS1=<mock-chroot> \s-\v\$  --setenv=LANG=C.UTF-8 --resolv-conf=off bash --login -c /usr/bin/rpmbuild -bb --target x86_64 --nodeps /builddir/build/SPECS/samba.spec


Version-Release number of selected component (if applicable):
rpm-4.14.3-14.el8_4.x86_64
mock-2.11-1.el8.noarch


How reproducible:
Always

Steps to Reproduce:
1. Have a CentOS 8 Linux machine
2. Use SPEC file from the following link to build RPMs with fedora-34-x86_64 mock config
https://github.com/gluster/samba-integration/blob/samba-build/packaging/samba-master.spec.j2


Actual results:
Failed in "Processing files" for python3-samba sub-package with the error mentioned in Description. See attachment for complete build log.

Expected results:
RPMs are built without any error.


Additional info:
The very same SPEC file builds successfully for fedora-33-x86_64 mock config

Comment 1 Anoop C S 2021-08-03 05:57:54 UTC
Do I have to provide anything extra as debug information to investigate the issue?

Comment 2 Florian Festi 2021-08-09 12:38:46 UTC
This issue is probably cause by there being two copies of a file which share the same build id (aka are identical) - one of the copies being /usr/lib64/python\
3.9/site-packages/samba/dsdb.cpython-39-x86_64-linux-gnu.so

The failing symlink is trying to point from the buildid (think hash of the file) to the two locations at once.

This can be avoided by getting rid of one of the copies - e.g. by replacing it with a symlink the the other copy - or by disabling debuginfo generation. The later one preventing debugging for the contents of the package and other programs using the libraries - obviously.

Comment 4 Michal Domonkos 2021-08-10 15:31:36 UTC
It turns out the root cause here is an old, known issue in RPM's build-id generation logic, more details here:
https://bugzilla.redhat.com/show_bug.cgi?id=878863

Basically, when using %exclude on files that are not packaged in any other subpackage in the same .spec file, RPM won't complain about unpackaged files and fail as it normally would, however the build-id generator will get confused and produce unexpected results.

In my testing with a simple spec file that contains:

%files
%{_bindir}/foo
%exclude %{_bindir}/foo

rpmbuild printed a warning saying "File listed twice: /usr/lib/.build-id/a1/e0c933eef9fa99e3b98ac6c42a927179846281" and the build finished successfully.  However, inspecting the binary package, this filename was a dangling symlink (because /usr/bin/foo had been excluded).

In the case of this samba package, however, the build failed with the "failed symlink: File exists" error; that could indicate that the build-id generator didn't detect the duplicate and just went ahead to process the same file twice.

I'm not quite sure why in the samba case the duplicate wasn't recognized, however I'm pretty sure the root cause was the same as in the previous example.

Even though the Bug 878863 is CLOSED, it's a valid defect in RPM which we're tracking upstream:
https://github.com/rpm-software-management/rpm/issues/1622

Either way, the "fix" here is to simply not use %exclude on unpackaged files in the .spec file [1].  After all, that's not how the %exclude directive is supposed to be used (see Panu's comments in Bug 878863).  Specifically, it's these two .so files:

%exclude %{python3_sitearch}/samba/dsdb.*.so
%exclude %{python3_sitearch}/samba/dsdb_dns.*.so

These are not packaged in any other subpackage, yet they remain the buildroot (triggering the above RPM bug), so to avoid this, you need to remove them in the %install section.  I tried to apply this change to the SRPM you provided above (thanks!) and it did resolve the problem for me.

That said, I'm closing this BZ as a DUPE.

[1] https://github.com/gluster/samba-integration/blob/samba-build/packaging/samba-master.spec.j2

*** This bug has been marked as a duplicate of bug 878863 ***

Comment 5 Michal Domonkos 2021-08-10 15:35:27 UTC
Just a side note - I did a git blame on the above spec file and it seems the following commit introduced this issues:
https://github.com/gluster/samba-integration/commit/fe4d4d02491ab0ee892cf97c69fcd66964a4aa19

Comment 6 Michal Domonkos 2021-08-10 16:48:19 UTC
Another side note:

As part of the debugging session, Anoop also pointed out to me that with the fedora-33 mock config, the build actually passes just fine - so in the hopes of understanding this issue better, I tried that... and indeed, it worked!  However, I got the following warnings, which confirm the suspected root cause described above:

warning: File listed twice: /usr/lib/.build-id/81/ef29a63209c5d5940f8d20cf46b5079b6a24f9
warning: File listed twice: /usr/lib/.build-id/9b/dc1e42c3fb842a08d070cabb45e297b2891ce8