Description of problem: I currently have both pypy3.7-libs and pypy3.8-libs installed. dnf wants to update pypy3.7-libs to its latest version, but this causes file conflicts on the test transaction. Version-Release number of selected component (if applicable): Existing packages are: $ rpm -q pypy3.7-libs pypy3.7-libs-7.3.6-1.fc34.x86_64 $ rpm -q pypy3.8-libs pypy3.8-libs-7.3.7-1.fc34.x86_64 The update is to: pypy3.7-libs-7.3.7-1.fc34.x86_64 Steps to Reproduce: Mock also works to reproduce: 1. mock -r fedora-34-x86_64 --install pypy3.7-libs pypy3.8-libs Actual results: ================================================================================================= Package Architecture Version Repository Size ================================================================================================= pypy3.7 x86_64 7.3.7-1.fc34 updates 11 M pypy3.7-devel x86_64 7.3.7-1.fc34 updates 59 k pypy3.7-libs x86_64 7.3.7-1.fc34 updates 13 M Transaction Summary ================================================================================================= Upgrade 3 Packages Skip 6 Packages Total download size: 24 M Is this ok [y/N]: y Downloading Packages: ... Running transaction check Transaction check succeeded. Running transaction test The downloaded packages were saved in cache until the next successful transaction. You can remove cached packages by executing 'dnf clean packages'. Error: Transaction test error: file /usr/lib/.build-id/39/208b4f57aa4d5cfcbed13cf6c5ec428de27264 from install of pypy3.7-libs-7.3.7-1.fc34.x86_64 conflicts with file from package pypy3.8-libs-7.3.7-1.fc34.x86_64 file /usr/lib/.build-id/40/682aae1fc61eaa03230eac9033407ce96488c9 from install of pypy3.7-libs-7.3.7-1.fc34.x86_64 conflicts with file from package pypy3.8-libs-7.3.7-1.fc34.x86_64 file /usr/lib/.build-id/53/96df0ea06a9dd7e8eeea66aaa65dda9b64e73c from install of pypy3.7-libs-7.3.7-1.fc34.x86_64 conflicts with file from package pypy3.8-libs-7.3.7-1.fc34.x86_64 file /usr/lib/.build-id/87/535f93c815a75349f4d56210c31d4ef4b797f2 from install of pypy3.7-libs-7.3.7-1.fc34.x86_64 conflicts with file from package pypy3.8-libs-7.3.7-1.fc34.x86_64 file /usr/lib/.build-id/c6/bff4486941a2fb3490a80eb7b7bfe3370e8ba0 from install of pypy3.7-libs-7.3.7-1.fc34.x86_64 conflicts with file from package pypy3.8-libs-7.3.7-1.fc34.x86_64 file /usr/lib/.build-id/c9/e94d839699198b0e87c334f8e03160fec054de from install of pypy3.7-libs-7.3.7-1.fc34.x86_64 conflicts with file from package pypy3.8-libs-7.3.7-1.fc34.x86_64 file /usr/lib/.build-id/e4/a09cf0647ed5d0e730dcdfef7aff9f9a2e11a0 from install of pypy3.7-libs-7.3.7-1.fc34.x86_64 conflicts with file from package pypy3.8-libs-7.3.7-1.fc34.x86_64 file /usr/lib/.build-id/e8/1db0d87e7cce4f5fcad18063f7b7fce2eb9590 from install of pypy3.7-libs-7.3.7-1.fc34.x86_64 conflicts with file from package pypy3.8-libs-7.3.7-1.fc34.x86_64 file /usr/lib/.build-id/ee/d5ab1faa168044b871001c4117fc8b7853b3c6 from install of pypy3.7-libs-7.3.7-1.fc34.x86_64 conflicts with file from package pypy3.8-libs-7.3.7-1.fc34.x86_64 Expected results: Update goes through without issue.
Funny business. Need to figure out why is that :( Some of the sources are identical, but nothing should be pre-built.
This seems to be Fedora 34 specific and affect multiple architectures: $ mock -r fedora-34-x86_64 install 'pypy3*-libs' ... Error: Transaction test error: file /usr/lib/.build-id/39/208b4f57aa4d5cfcbed13cf6c5ec428de27264 conflicts between attempted installs of pypy3.8-libs-7.3.7-1.fc34.x86_64 and pypy3.7-libs-7.3.7-1.fc34.x86_64 file /usr/lib/.build-id/40/682aae1fc61eaa03230eac9033407ce96488c9 conflicts between attempted installs of pypy3.8-libs-7.3.7-1.fc34.x86_64 and pypy3.7-libs-7.3.7-1.fc34.x86_64 file /usr/lib/.build-id/53/96df0ea06a9dd7e8eeea66aaa65dda9b64e73c conflicts between attempted installs of pypy3.8-libs-7.3.7-1.fc34.x86_64 and pypy3.7-libs-7.3.7-1.fc34.x86_64 file /usr/lib/.build-id/87/535f93c815a75349f4d56210c31d4ef4b797f2 conflicts between attempted installs of pypy3.8-libs-7.3.7-1.fc34.x86_64 and pypy3.7-libs-7.3.7-1.fc34.x86_64 file /usr/lib/.build-id/c6/bff4486941a2fb3490a80eb7b7bfe3370e8ba0 conflicts between attempted installs of pypy3.8-libs-7.3.7-1.fc34.x86_64 and pypy3.7-libs-7.3.7-1.fc34.x86_64 file /usr/lib/.build-id/c9/e94d839699198b0e87c334f8e03160fec054de conflicts between attempted installs of pypy3.8-libs-7.3.7-1.fc34.x86_64 and pypy3.7-libs-7.3.7-1.fc34.x86_64 file /usr/lib/.build-id/e4/a09cf0647ed5d0e730dcdfef7aff9f9a2e11a0 conflicts between attempted installs of pypy3.8-libs-7.3.7-1.fc34.x86_64 and pypy3.7-libs-7.3.7-1.fc34.x86_64 file /usr/lib/.build-id/e8/1db0d87e7cce4f5fcad18063f7b7fce2eb9590 conflicts between attempted installs of pypy3.8-libs-7.3.7-1.fc34.x86_64 and pypy3.7-libs-7.3.7-1.fc34.x86_64 file /usr/lib/.build-id/ee/d5ab1faa168044b871001c4117fc8b7853b3c6 conflicts between attempted installs of pypy3.8-libs-7.3.7-1.fc34.x86_64 and pypy3.7-libs-7.3.7-1.fc34.x86_64 $ mock -r fedora-34-ppc64le install 'pypy3*-libs' ... Error: Transaction test error: file /usr/lib/.build-id/11/291d77c45fbf4158f080470907a950b3cae8bc conflicts between attempted installs of pypy3.8-libs-7.3.7-1.fc34.ppc64le and pypy3.7-libs-7.3.7-1.fc34.ppc64le file /usr/lib/.build-id/19/71751cfb824c91ef0746b9adc7ffd659c0660f conflicts between attempted installs of pypy3.8-libs-7.3.7-1.fc34.ppc64le and pypy3.7-libs-7.3.7-1.fc34.ppc64le file /usr/lib/.build-id/1a/5f6f8ec36564bfe757782e0fe201b48e253f81 conflicts between attempted installs of pypy3.8-libs-7.3.7-1.fc34.ppc64le and pypy3.7-libs-7.3.7-1.fc34.ppc64le file /usr/lib/.build-id/1e/66cd8287c61888949198f37bc797c75738a7ec conflicts between attempted installs of pypy3.8-libs-7.3.7-1.fc34.ppc64le and pypy3.7-libs-7.3.7-1.fc34.ppc64le file /usr/lib/.build-id/49/edf504ef8678601ac7cd30eb6d3b43cce582a9 conflicts between attempted installs of pypy3.8-libs-7.3.7-1.fc34.ppc64le and pypy3.7-libs-7.3.7-1.fc34.ppc64le file /usr/lib/.build-id/55/4603e5db36f5a6ab98cfbf0dd732df71a498bc conflicts between attempted installs of pypy3.8-libs-7.3.7-1.fc34.ppc64le and pypy3.7-libs-7.3.7-1.fc34.ppc64le file /usr/lib/.build-id/5f/37e5749692888f552bc26c30cb7776407cd002 conflicts between attempted installs of pypy3.8-libs-7.3.7-1.fc34.ppc64le and pypy3.7-libs-7.3.7-1.fc34.ppc64le file /usr/lib/.build-id/79/91da4a2d4a81bb9f98a3f6417d79ca33c54782 conflicts between attempted installs of pypy3.8-libs-7.3.7-1.fc34.ppc64le and pypy3.7-libs-7.3.7-1.fc34.ppc64le file /usr/lib/.build-id/b2/9a6ac055682e3946da32adee5f3337787a778a conflicts between attempted installs of pypy3.8-libs-7.3.7-1.fc34.ppc64le and pypy3.7-libs-7.3.7-1.fc34.ppc64le $ mock -r fedora-35-x86_64 install 'pypy3*-libs' ... Installed: pypy3.7-libs-7.3.7-1.fc35.x86_64 pypy3.8-7.3.7-1.fc35.x86_64 Same for 36/37.
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/LEAEQW3KOXDGDZC2SDIYISB3ZBCKTNJJ/ https://lists.fedoraproject.org/archives/list/packaging@lists.fedoraproject.org/thread/LEAEQW3KOXDGDZC2SDIYISB3ZBCKTNJJ/
For reference, one can use the fedora debuginfod service to decode which particular libraries are conflicting. For example, in the case of file /usr/lib/.build-id/c9/e94d839699198b0e87c334f8e03160fec054de from install of pypy3.7-libs-7.3.7-1.fc34.x86_64 conflicts with file from package pypy3.8-libs-7.3.7-1.fc34.x86_64 use DEBUGINFOD_URLS=https://debuginfod.fedoraproject.org/ debuginfod-find -v debuginfo c9e94d839699198b0e87c334f8e03160fec054de to see (inter alia) one of the matches: x-debuginfod-file: /usr/lib/debug/usr/lib64/pypy3.8/lib/pypy3.8/_pwdgrp_cffi.pypy38-pp73-x86_64-linux-gnu.so-7.3.7-1.fc34.x86_64.debug x-debuginfod-archive: /mnt/fedora_koji_prod/koji/packages/pypy3.8/7.3.7/1.fc34/x86_64/pypy3.8-libs-debuginfo-7.3.7-1.fc34.x86_64.rpm x-debuginfod-size: 5512
https://src.fedoraproject.org/rpms/pypy3.8/pull-request/3 https://src.fedoraproject.org/rpms/pypy3.7/pull-request/16 If it builds, I'll test it on Fedora 34.
Could we tweak find-debuginfo to default --build-id-seed from --unique-debug-prefix / --unique-debug-src-base ?
This would be another victim of modularity. I think %{name} is quite intentionally excluded from the build-id seed, but that was before we had all this content where we had all this intentionally parallel content in the distro. This is probably a fairly special case even in the world of modularity but it's also a legit use and thus rpm needs to permit that somehow. It'd be simple enough to either a) give up and add the name to the seed b) macroize the buildid seed that these specific cases can override it Mark, thoughts?
Note that in https://src.fedoraproject.org/rpms/pypy3.8/pull-request/3 and https://src.fedoraproject.org/rpms/pypy3.7/pull-request/16 I tried to change the seed but it fails the build and I don't know why. Details: https://src.fedoraproject.org/rpms/pypy3.7/pull-request/16#comment-97417
I haven't investigated yet where the duplicate buildid comes from in this case, sorry. Adding the name to the seed should be fine. But also really shouldn't be necessary. In general buildids only clash when there are binaries which aren't rebuild or copied as is. Or if something is build without debuginfo. Because when there is debuginfo that includes the compiler version and flags plus the (unique) builddir (which already includes the name) and all that info is included in the globally unique hash. So I think we really should investigate which binaries have duplicate buildids and if they are simply copies or if they have been build without debuginfo.
The duplicate build-id comes from the fact that between pypy3.7 and pypy3.8 some files are identical and the version and release is identical. builddir is not included in the binaries, is it?
(In reply to Miro Hrončok from comment #10) > The duplicate build-id comes from the fact that between pypy3.7 and pypy3.8 > some files are identical and the version and release is identical. builddir > is not included in the binaries, is it? The builddir should be included in the debuginfo, which is used to generate the buildid (before it is stripped into a separate file).
> The builddir should be included in the debuginfo .... but pypy sometimes invokes an inferior gcc without CFLAGS=-g, as do other tools (!). So we lack debuginfo for those artifacts, so the build path in the dwarf is not there to disambiguate the builds.
(In reply to Panu Matilainen from comment #7) > This would be another victim of modularity. I think %{name} is quite > intentionally excluded from the build-id seed, but that was before we had > all this content where we had all this intentionally parallel content in the > distro. I don't understand how modular packages would interfere. At the worst case, we may have some duplicate debuginfo content with different buildids. That's not bad at all - they don't conflict, and modern stuff like debuginfod will let consumers take only what they need. IOW, non-conflict is far more important than space savings.
Never mind my rambling about modularity, seems I missed some finer nuances here.
(In reply to Frank Ch. Eigler from comment #12) > > The builddir should be included in the debuginfo > > .... but pypy sometimes invokes an inferior gcc without CFLAGS=-g, as do > other tools (!). So we lack debuginfo for those artifacts, so the build > path in the dwarf is not there to disambiguate the builds. Cool find. Can you point me to the invocations so I can try to fix them?
> > .... but pypy sometimes invokes an inferior gcc without CFLAGS=-g, as do > > other tools (!). So we lack debuginfo for those artifacts, so the build > > path in the dwarf is not there to disambiguate the builds. > > Cool find. Can you point me to the invocations so I can try to fix them? The pypy files in question are cases in point. (The _pwdgrp_cffi.pypy38-pp73-x86_64-linux-gnu.so-7.3.7-1.fc34.x86_64.debug file has no .debug_info* sections.) While a local fix for this may be beneficial, I believe it'd be ALSO good to get /usr/bin/find-debuginfo to add a seed by default. I'll gather some stats about fedora buildids from the debuginfod server index to see how widespread this problem is.
(In reply to Frank Ch. Eigler from comment #16) > While a local fix for this may be beneficial, I believe it'd be ALSO good > to get /usr/bin/find-debuginfo to add a seed by default. I'll gather some > stats about fedora buildids from the debuginfod server index to see how > widespread this problem is. We could add a warning to find-debuginfo.sh in case it finds an buildid in a file that doesn't have debuginfo. And maybe make the rpm install warnings a little easier to interpret (those conflicting files are symlinks, the target is the binary with the conflicting build-id). So if the warning could (also) print the target of that symlink that would be really helpful. Adding an extra, known to really be different, seed to the build-id calculation would mean we miss these bugs. Unless these kind of bugs are widespread and really hard to fix.
(In reply to Mark Wielaard from comment #17) > Adding an extra, known to really be different, seed to the build-id > calculation would mean we miss these bugs. Well, not really "bugs", just peculiarities. > Unless these kind of bugs are widespread and really hard to fix. I have some data about this, and it's kind of ugly. Across all of fedora 32+, there are some 370,000 duplicate buildids between debuginfo-carrying files in different rpms or different files in the same rpm. That is so many that it's not practical to even analyze all the cases. Just a random set of categories: (a) duplicate binaries across different n-v-r's - glibc /usr/lib/gconv/*.so binaries identical across several releases - glibc benchtests binaries, ditto - pypy3.7 and pypy3.8 so's identical - gcc /usr/libexec/.../vet - hdf libmfhdf.so - cheese .dwz files (but not binaries!) - dyninst tests (b) not-really-problems: subpackages from a single n-v-r build duplicating binaries under different names - java subpackages include dupe binaries in different subdirs - crossgcc holy cow so many identical binaries for different targets - graphviz including php/tcl .so's in different subdirs - quantum-espresso with identical /usr/bin/*.x_binary files - charliecloud test .sos copied with multiple prefixes (not hardlinked) - binutils ld & ld.gold (copied, not hardlinked) (c) rust binaries - it seems as though stripping / -g compilation works differently enough that debuginfod considers /usr/bin/FOO and /usr/lib/debug/usr/bin/FOO.debug the same| (d) non-problems - same binary included in more than one subrpm from a build (e) OTHER UNKNOWN SITUATIONS that I didn't notice, because the report is so big Of these, the (a) case could be fixed with automatic n-v-r.a based buildid salting.
Created attachment 1861767 [details] REPORT.txt.gz of duplicate buildids, with source-rpm (path) and installed-file name
OK, there seem to be no quick and correct way to fix this, so I'll include the Python version in the release.
Included in: https://src.fedoraproject.org/rpms/pypy3.7/pull-request/17 + 18 + 19 + 20 https://src.fedoraproject.org/rpms/pypy3.8/pull-request/4 + 5 + 6 + 7 https://src.fedoraproject.org/rpms/pypy3.9/pull-request/4 + 5 + 6 + 7
I've verified that pypy3.X-libs-7.3.8-1.3.X.fc34.x86_64 for X={7,8,9} is co-installable on Fedora 34.
FEDORA-2022-b8fae1df9b has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2022-b8fae1df9b
FEDORA-2022-b8fae1df9b has been pushed to the Fedora 37 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2022-e562c1915a has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2022-e562c1915a
FEDORA-2022-00cbb41d9e has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-00cbb41d9e
FEDORA-2022-fc95b57dfe has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-fc95b57dfe
FEDORA-2022-6cdfbb7f54 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-6cdfbb7f54
FEDORA-2022-0752b72aed has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2022-0752b72aed
FEDORA-2022-00cbb41d9e has been pushed to the Fedora 35 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-00cbb41d9e` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-00cbb41d9e See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2022-fc95b57dfe has been pushed to the Fedora 35 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-fc95b57dfe` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-fc95b57dfe See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2022-6cdfbb7f54 has been pushed to the Fedora 35 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-6cdfbb7f54` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-6cdfbb7f54 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2022-e562c1915a has been pushed to the Fedora 34 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-e562c1915a` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-e562c1915a See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2022-4e964e4b49 has been pushed to the Fedora 34 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-4e964e4b49` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-4e964e4b49 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2022-0752b72aed has been pushed to the Fedora 34 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-0752b72aed` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-0752b72aed See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2022-00cbb41d9e has been pushed to the Fedora 35 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2022-fc95b57dfe has been pushed to the Fedora 35 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2022-6cdfbb7f54 has been pushed to the Fedora 35 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2022-e562c1915a has been pushed to the Fedora 34 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2022-4e964e4b49 has been pushed to the Fedora 34 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2022-0752b72aed has been pushed to the Fedora 34 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2023-c729dabeb1 has been submitted as an update to Fedora 40. https://bodhi.fedoraproject.org/updates/FEDORA-2023-c729dabeb1
FEDORA-2023-c729dabeb1 has been pushed to the Fedora 40 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2023-ddde191e04 has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-ddde191e04
FEDORA-2023-ddde191e04 has been pushed to the Fedora 39 stable repository. If problem still persists, please make note of it in this bug report.