Bug 1430587 - rpm 4.13.0.1-4.fc27 breaks powerpc64 kernel builds
Summary: rpm 4.13.0.1-4.fc27 breaks powerpc64 kernel builds
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: rpm
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Mark Wielaard
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1430764 (view as bug list)
Depends On:
Blocks: 1430565
TreeView+ depends on / blocked
 
Reported: 2017-03-09 01:48 UTC by Laura Abbott
Modified: 2017-03-10 12:09 UTC (History)
7 users (show)

Fixed In Version: 4.13.0.1-7
Clone Of:
Environment:
Last Closed: 2017-03-10 12:09:32 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Laura Abbott 2017-03-09 01:48:00 UTC
kernel rawhide builds for powerpc64 no longer build. From https://koji.fedoraproject.org/koji/getfile?taskID=18270481&name=build.log&offset=-4000

Wrote: /builddir/build/RPMS/kernel-4.11.0-0.rc1.git0.2.fc27.ppc64le.rpm
Wrote: /builddir/build/RPMS/kernel-headers-4.11.0-0.rc1.git0.2.fc27.ppc64le.rpm
Wrote: /builddir/build/RPMS/kernel-cross-headers-4.11.0-0.rc1.git0.2.fc27.ppc64le.rpm
error: Bad CSA data
RPM build errors:
    Missing build-id in /builddir/build/BUILDROOT/kernel-4.11.0-0.rc1.git0.2.fc27.ppc64le/usr/lib64/kernel-wrapper/pseries-head.o
    Missing build-id in /builddir/build/BUILDROOT/kernel-4.11.0-0.rc1.git0.2.fc27.ppc64le/usr/lib64/kernel-wrapper/epapr-wrapper.o
    Missing build-id in /builddir/build/BUILDROOT/kernel-4.11.0-0.rc1.git0.2.fc27.ppc64le/usr/lib64/kernel-wrapper/empty.o
    Missing build-id in /builddir/build/BUILDROOT/kernel-4.11.0-0.rc1.git0.2.fc27.ppc64le/usr/lib64/kernel-wrapper/of.o
    Missing build-id in /builddir/build/BUILDROOT/kernel-4.11.0-0.rc1.git0.2.fc27.ppc64le/usr/lib64/kernel-wrapper/epapr.o
    Duplicate build-ids /builddir/build/BUILDROOT/kernel-4.11.0-0.rc1.git0.2.fc27.ppc64le/lib/modules/4.11.0-0.rc1.git0.2.fc27.ppc64le/vmlinuz and /builddir/build/BUILDROOT/kernel-4.11.0-0.rc1.git0.2.fc27.ppc64le/boot/vmlinuz-4.11.0-0.rc1.git0.2.fc27.ppc64le
    Missing build-id in /builddir/build/BUILDROOT/kernel-4.11.0-0.rc1.git0.2.fc27.ppc64le/usr/src/kernels/4.11.0-0.rc1.git0.2.fc27.ppc64le/arch/powerpc/lib/crtsavres.o
    Bad CSA data

Kernels that previously built under 4.13.0.1-3.fc26 now fail under 4.13.0.1-4.fc27. The commits in http://pkgs.fedoraproject.org/cgit/rpms/rpm.git/commit/?id=5b4d9835cd5259214ad957e26e32b97926630837 also seems to be related to buildids. I haven't narrowed down which patch is causing build failures but I'm happy to provide more kernel information if necessary.

Comment 1 Mark Wielaard 2017-03-09 07:59:00 UTC
Thanks for the report. I'll try to see if I can find a ppc64le setup to replicate.

The Missing build-id messages seem bogus since they are for .o object files and those aren't supposed to have build-ids (only loadable ELF images should, like executables, shared libaries or kernel modules).

Are the vmlinuz and vmlinuz-4.11.0-0.rc1.git0.2.fc27.ppc64le files hard linked? Then the Duplicate build-id message is also bogus.

The Bad CSA data message indicate that no files were included in the cpioList (file list) of the package. Which might or might not be related to the build-id warnings.

Comment 2 Mark Wielaard 2017-03-09 08:22:29 UTC
I think this rpm patch might solve the issue:

diff --git a/build/files.c b/build/files.c
index cbcc0a9..26f53bd 100644
--- a/build/files.c
+++ b/build/files.c
@@ -1709,8 +1709,10 @@ static int generateBuildIDs(FileList fl)
 	if (lstat(flp->diskPath, &sbuf) == 0 && S_ISREG (sbuf.st_mode)) {
 	    int fd = open (flp->diskPath, O_RDONLY);
 	    if (fd >= 0) {
+		GElf_Ehdr ehdr;
 		Elf *elf = elf_begin (fd, ELF_C_READ, NULL);
-		if (elf != NULL && elf_kind(elf) == ELF_K_ELF) {
+		if (elf != NULL && elf_kind(elf) == ELF_K_ELF
+		    && gelf_getehdr(elf, &ehdr) != NULL) {
 		    const void *build_id;
 		    ssize_t len = dwelf_elf_gnu_build_id (elf, &build_id);
 		    /* len == -1 means error. Zero means no
@@ -1759,9 +1761,14 @@ static int generateBuildIDs(FileList fl)
 				   _("error reading build-id in %s: %s\n"),
 				   flp->diskPath, elf_errmsg (-1));
 			} else if (len == 0) {
-			    rpmlog(terminate ? RPMLOG_ERR : RPMLOG_WARNING,
-				   _("Missing build-id in %s\n"),
-				   flp->diskPath);
+			    /* Only ET_EXEC, ET_DYN or kernel modules
+			       have build-ids. */
+			    if (ehdr.e_type == ET_EXEC || ehdr.e_type == ET_DYN
+				|| (ehdr.e_type == ET_REL
+				    && rpmFileHasSuffix (flp->diskPath, ".ko")))
+			      rpmlog(terminate ? RPMLOG_ERR : RPMLOG_WARNING,
+				     _("Missing build-id in %s\n"),
+				     flp->diskPath);
 			} else {
 			    rpmlog(terminate ? RPMLOG_ERR : RPMLOG_WARNING,
 				   (len < 16

I'll see if I can find a way to test this on a ppc64le setup.

Comment 3 Igor Gnatenko 2017-03-09 10:37:08 UTC
Better, but not enough:
    Duplicate build-ids /builddir/build/BUILDROOT/kernel-4.11.0-0.rc1.git1.1.fc27.ppc64le/lib/modules/4.11.0-0.rc1.git1.1.fc27.ppc64le/vmlinuz and /builddir/build/BUILDROOT/kernel-4.11.0-0.rc1.git1.1.fc27.ppc64le/boot/vmlinuz-4.11.0-0.rc1.git1.1.fc27.ppc64le
    Duplicate build-ids /builddir/build/BUILDROOT/kernel-4.11.0-0.rc1.git1.1.fc27.ppc64le/usr/lib/debug/usr/lib/debug/lib/modules/4.11.0-0.rc1.git1.1.fc27.ppc64le/vmlinux.debug and /builddir/build/BUILDROOT/kernel-4.11.0-0.rc1.git1.1.fc27.ppc64le/usr/lib/debug/lib/modules/4.11.0-0.rc1.git1.1.fc27.ppc64le/vmlinux
    create archive failed: cpio: (error 0xffffffff)

Comment 4 Mark Wielaard 2017-03-09 10:56:09 UTC
So that patch definitely helped to get rid of those missing build-id errors.
And it has already been integrated upstream.

But there is still something odd going on. See https://copr-be.cloud.fedoraproject.org/results/ignatenkobrain/debuginfo-kernel-ppc64le/fedora-rawhide-ppc64le/00523940-kernel/build.log.gz

Processing files: kernel-debuginfo-4.11.0-0.rc1.git1.1.fc27.ppc64le
warning: Duplicate build-ids /builddir/build/BUILDROOT/kernel-4.11.0-0.rc1.git1.1.fc27.ppc64le/usr/lib/debug/usr/lib/debug/lib/modules/4.11.0-0.rc1.git1.1.fc27.ppc64le/vmlinux.debug and /builddir/build/BUILDROOT/kernel-4.11.0-0.rc1.git1.1.fc27.ppc64le/usr/lib/debug/lib/modules/4.11.0-0.rc1.git1.1.fc27.ppc64le/vmlinux
Provides: installonlypkg(kernel) kernel-debuginfo = 4.11.0-0.rc1.git1.1.fc27 kernel-debuginfo(ppc-64) = 4.11.0-0.rc1.git1.1.fc27 kernel-debuginfo-ppc64le = 4.11.0-0.rc1.git1.1.fc27
Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1
Checking for unpackaged file(s): /usr/lib/rpm/check-files /builddir/build/BUILDROOT/kernel-4.11.0-0.rc1.git1.1.fc27.ppc64le
Wrote: /builddir/build/RPMS/kernel-4.11.0-0.rc1.git1.1.fc27.ppc64le.rpm
Wrote: /builddir/build/RPMS/kernel-headers-4.11.0-0.rc1.git1.1.fc27.ppc64le.rpm
Wrote: /builddir/build/RPMS/kernel-cross-headers-4.11.0-0.rc1.git1.1.fc27.ppc64le.rpm
error: create archive failed: cpio: (error 0xffffffff)
RPM build errors:
    line 903: Possible unexpanded macro in: %package %{pae}-core
    line 903: Possible unexpanded macro in: Provides: kernel-%{pae}-core-uname-r = 4.11.0-0.rc1.git1.1.fc27.ppc64le+%{pae}
    line 903: Possible unexpanded macro in: Provides: kernel-ppc64le = 4.11.0-0.rc1.git1.1.fc27+%{pae}
    line 903: Possible unexpanded macro in: Provides: kernel-uname-r = 4.11.0-0.rc1.git1.1.fc27.ppc64le+%{pae}
    line 903: Possible unexpanded macro in: %package %{pae}
    line 903: Possible unexpanded macro in: Requires: kernel-%{pae}-core-uname-r = 4.11.0-0.rc1.git1.1.fc27.ppc64le+%{pae}
    line 903: Possible unexpanded macro in: Requires: kernel-%{pae}-modules-uname-r = 4.11.0-0.rc1.git1.1.fc27.ppc64le+%{pae}
    line 903: Possible unexpanded macro in: %package %{pae}-devel
    line 903: Possible unexpanded macro in: Provides: kernel-devel-ppc64le = 4.11.0-0.rc1.git1.1.fc27+%{pae}
    line 903: Possible unexpanded macro in: Provides: kernel-devel-uname-r = 4.11.0-0.rc1.git1.1.fc27.ppc64le+%{pae}
    line 903: Possible unexpanded macro in: %package %{pae}-modules
    line 903: Possible unexpanded macro in: Provides: kernel-modules-ppc64le = 4.11.0-0.rc1.git1.1.fc27+%{pae}
    line 903: Possible unexpanded macro in: Provides: kernel-modules = 4.11.0-0.rc1.git1.1.fc27+%{pae}
    line 903: Possible unexpanded macro in: Provides: kernel-%{pae}-modules-uname-r = 4.11.0-0.rc1.git1.1.fc27.ppc64le+%{pae}
    line 903: Possible unexpanded macro in: Requires: kernel-uname-r = 4.11.0-0.rc1.git1.1.fc27.ppc64le+%{pae}
    line 903: Possible unexpanded macro in: %package %{pae}-modules-extra
    line 903: Possible unexpanded macro in: Provides: kernel-%{pae}-modules-extra-ppc64le = 4.11.0-0.rc1.git1.1.fc27+%{pae}
    line 903: Possible unexpanded macro in: Provides: kernel-%{pae}-modules-extra = 4.11.0-0.rc1.git1.1.fc27+%{pae}
    line 903: Possible unexpanded macro in: Provides: kernel-%{pae}-modules-extra-uname-r = 4.11.0-0.rc1.git1.1.fc27.ppc64le+%{pae}
    line 903: Possible unexpanded macro in: Requires: kernel-uname-r = 4.11.0-0.rc1.git1.1.fc27.ppc64le+%{pae}
    line 903: Possible unexpanded macro in: Requires: kernel-%{pae}-modules-uname-r = 4.11.0-0.rc1.git1.1.fc27.ppc64le+%{pae}
    line 903: Possible unexpanded macro in: %package %{pae}-debuginfo
    line 919: Possible unexpanded macro in: %package %{pae}debug-core
    line 919: Possible unexpanded macro in: Provides: kernel-%{pae}debug-core-uname-r = 4.11.0-0.rc1.git1.1.fc27.ppc64le+%{pae}debug
    line 919: Possible unexpanded macro in: Provides: kernel-ppc64le = 4.11.0-0.rc1.git1.1.fc27+%{pae}debug
    line 919: Possible unexpanded macro in: Provides: kernel-uname-r = 4.11.0-0.rc1.git1.1.fc27.ppc64le+%{pae}debug
    line 919: Possible unexpanded macro in: %package %{pae}debug
    line 919: Possible unexpanded macro in: Requires: kernel-%{pae}debug-core-uname-r = 4.11.0-0.rc1.git1.1.fc27.ppc64le+%{pae}debug
    line 919: Possible unexpanded macro in: Requires: kernel-%{pae}debug-modules-uname-r = 4.11.0-0.rc1.git1.1.fc27.ppc64le+%{pae}debug
    line 919: Possible unexpanded macro in: %package %{pae}debug-devel
    line 919: Possible unexpanded macro in: Provides: kernel-devel-ppc64le = 4.11.0-0.rc1.git1.1.fc27+%{pae}debug
    line 919: Possible unexpanded macro in: Provides: kernel-devel-uname-r = 4.11.0-0.rc1.git1.1.fc27.ppc64le+%{pae}debug
    line 919: Possible unexpanded macro in: %package %{pae}debug-modules
    line 919: Possible unexpanded macro in: Provides: kernel-modules-ppc64le = 4.11.0-0.rc1.git1.1.fc27+%{pae}debug
    line 919: Possible unexpanded macro in: Provides: kernel-modules = 4.11.0-0.rc1.git1.1.fc27+%{pae}debug
    line 919: Possible unexpanded macro in: Provides: kernel-%{pae}debug-modules-uname-r = 4.11.0-0.rc1.git1.1.fc27.ppc64le+%{pae}debug
    line 919: Possible unexpanded macro in: Requires: kernel-uname-r = 4.11.0-0.rc1.git1.1.fc27.ppc64le+%{pae}debug
    line 919: Possible unexpanded macro in: %package %{pae}debug-modules-extra
    line 919: Possible unexpanded macro in: Provides: kernel-%{pae}debug-modules-extra-ppc64le = 4.11.0-0.rc1.git1.1.fc27+%{pae}debug
    line 919: Possible unexpanded macro in: Provides: kernel-%{pae}debug-modules-extra = 4.11.0-0.rc1.git1.1.fc27+%{pae}debug
    line 919: Possible unexpanded macro in: Provides: kernel-%{pae}debug-modules-extra-uname-r = 4.11.0-0.rc1.git1.1.fc27.ppc64le+%{pae}debug
    line 919: Possible unexpanded macro in: Requires: kernel-uname-r = 4.11.0-0.rc1.git1.1.fc27.ppc64le+%{pae}debug
    line 919: Possible unexpanded macro in: Requires: kernel-%{pae}debug-modules-uname-r = 4.11.0-0.rc1.git1.1.fc27.ppc64le+%{pae}debug
    line 919: Possible unexpanded macro in: %package %{pae}debug-debuginfo
    Duplicate build-ids /builddir/build/BUILDROOT/kernel-4.11.0-0.rc1.git1.1.fc27.ppc64le/lib/modules/4.11.0-0.rc1.git1.1.fc27.ppc64le/vmlinuz and /builddir/build/BUILDROOT/kernel-4.11.0-0.rc1.git1.1.fc27.ppc64le/boot/vmlinuz-4.11.0-0.rc1.git1.1.fc27.ppc64le
    Duplicate build-ids /builddir/build/BUILDROOT/kernel-4.11.0-0.rc1.git1.1.fc27.ppc64le/usr/lib/debug/usr/lib/debug/lib/modules/4.11.0-0.rc1.git1.1.fc27.ppc64le/vmlinux.debug and /builddir/build/BUILDROOT/kernel-4.11.0-0.rc1.git1.1.fc27.ppc64le/usr/lib/debug/lib/modules/4.11.0-0.rc1.git1.1.fc27.ppc64le/vmlinux
    create archive failed: cpio: (error 0xffffffff)
Child return code was: 1

Comment 5 Mark Wielaard 2017-03-09 10:59:11 UTC
(In reply to Igor Gnatenko from comment #3)
> Better, but not enough:
>     Duplicate build-ids
> /builddir/build/BUILDROOT/kernel-4.11.0-0.rc1.git1.1.fc27.ppc64le/lib/
> modules/4.11.0-0.rc1.git1.1.fc27.ppc64le/vmlinuz and
> /builddir/build/BUILDROOT/kernel-4.11.0-0.rc1.git1.1.fc27.ppc64le/boot/
> vmlinuz-4.11.0-0.rc1.git1.1.fc27.ppc64le
>     Duplicate build-ids
> /builddir/build/BUILDROOT/kernel-4.11.0-0.rc1.git1.1.fc27.ppc64le/usr/lib/
> debug/usr/lib/debug/lib/modules/4.11.0-0.rc1.git1.1.fc27.ppc64le/vmlinux.
> debug and
> /builddir/build/BUILDROOT/kernel-4.11.0-0.rc1.git1.1.fc27.ppc64le/usr/lib/
> debug/lib/modules/4.11.0-0.rc1.git1.1.fc27.ppc64le/vmlinux
>     create archive failed: cpio: (error 0xffffffff)

I think those Duplicate build-ids are harmless since they should only be warnings. Although it would definitely be better if we understood why that happens.

The real error seems to the the cpio archive creation failure.

It is a pity we don't have access to a ppc64le kernel buildroot to better investigate what is going on.

Comment 6 Igor Gnatenko 2017-03-09 14:46:23 UTC
*** Bug 1430764 has been marked as a duplicate of this bug. ***

Comment 7 Mark Wielaard 2017-03-10 08:50:35 UTC
The patch in comment #2 was subtly wrong, but had the correct idea.
It was wrong because it suppressed the bad warning/error message, but then still failed... Making it harder to understand what was going on.

Followup patch that corrects the actual error posted, but hasn't shown up in the archives yet: 
http://lists.rpm.org/pipermail/rpm-maint/2017-March/date.html

With that both the ppc64 kernel package and the LibRaw both build without issue. In both cases it was triggered by a .o object file that was added to the binary package (and which was erroneously flagged for not having a build-id).

Comment 8 Mark Wielaard 2017-03-10 09:21:47 UTC
(In reply to Mark Wielaard from comment #7)
> The patch in comment #2 was subtly wrong, but had the correct idea.
> It was wrong because it suppressed the bad warning/error message, but then
> still failed... Making it harder to understand what was going on.
> 
> Followup patch that corrects the actual error posted, but hasn't shown up in
> the archives yet: 
> http://lists.rpm.org/pipermail/rpm-maint/2017-March/date.html

Sorry, it was. I had the archive upside down:

http://lists.rpm.org/pipermail/rpm-maint/2017-March/005233.html
And it has now also been applied upstream:
http://lists.rpm.org/pipermail/rpm-maint/2017-March/005235.html


Note You need to log in before you can comment on or make changes to this bug.