Bug 1405570 - koji build for rawhide fails with <class '_rpm.error'>: error reading package header
koji build for rawhide fails with <class '_rpm.error'>: error reading package...
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: rpm (Show other bugs)
26
aarch64 Unspecified
unspecified Severity urgent
: ---
: ---
Assigned To: Panu Matilainen
Fedora Extras Quality Assurance
https://koji.fedoraproject.org/koji/t...
: Reopened
Depends On:
Blocks: ARMTracker 1376324
  Show dependency treegraph
 
Reported: 2016-12-16 13:08 EST by Mattias Ellert
Modified: 2017-02-28 20:25 EST (History)
10 users (show)

See Also:
Fixed In Version: rpm-4.13.0-11.fc26 rpm-4.13.0.1-1.fc24 rpm-4.13.0.1-1.fc25
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-02-28 20:21:03 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Mattias Ellert 2016-12-16 13:08:31 EST
Description of problem:

I have now tried to build an update for Fedora Rawhide several times during the last week, and it fails consistently.

Build:
https://koji.fedoraproject.org/koji/buildinfo?buildID=823483

Tasks for the three attempts:
https://koji.fedoraproject.org/koji/taskinfo?taskID=16787974
https://koji.fedoraproject.org/koji/taskinfo?taskID=16796570
https://koji.fedoraproject.org/koji/taskinfo?taskID=16800637
https://koji.fedoraproject.org/koji/taskinfo?taskID=16856402
https://koji.fedoraproject.org/koji/taskinfo?taskID=16905568

For each of the attempts, the sub tasks of the srpm creation and the package builds for each architecture succeeds. There is however no tagBuild subtask started. When pressing the "Show results" link for the main task the same - to me very cryptic error message - is displayed for all of the attempts:

<class '_rpm.error'>: error reading package header

The same build for Fedora 25, Fedora 24 and EPEL 7 has competed without problems:

https://koji.fedoraproject.org/koji/buildinfo?buildID=823482
https://koji.fedoraproject.org/koji/buildinfo?buildID=823485
https://koji.fedoraproject.org/koji/buildinfo?buildID=823484

So the problem is Rawhide specific, and I can not tell what the cause is from the information I see en koji.

Version-Release number of selected component (if applicable):
The version on the Fedora Build system.

How reproducible:
Five times so far.

Steps to Reproduce:
1. See above

Actual results:
<class '_rpm.error'>: error reading package header

Expected results:
Successful build
Comment 1 Mike McLean 2016-12-16 14:18:33 EST
I doubt this is a koji bug.

Looks like broken deps in the buildroot

DEBUG util.py:426:  nothing provides libpoppler.so.65()(64bit) needed by texlive-xetex-bin-6:svn41091-24.20160520.fc26.1.x86_64
Comment 2 Mattias Ellert 2016-12-16 15:10:03 EST
You are looking a recent resubmit by kevin that suffers from a newly introduced problem that is unrelated to the reported bug. Please follow the links to the failed tasks in the bug report.
Comment 3 Mattias Ellert 2016-12-17 12:17:24 EST
Texlive's broken deps due to the poppler soname bump was fixed by a rebuild.

Trying building again gives the same error as before:

https://koji.fedoraproject.org/koji/taskinfo?taskID=16921481

class '_rpm.error'>: error reading package header
Comment 4 Mike McLean 2016-12-17 14:31:18 EST
One of the rpms is bad and does not read

# rpm -Kv /mnt/koji//work/tasks/1483/16921483/root-debuginfo-6.08.02-1.fc26.aarch64.rpm
error: /mnt/koji//work/tasks/1483/16921483/root-debuginfo-6.08.02-1.fc26.aarch64.rpm: headerRead failed: hdr magic: BAD

This is the only rpm in the set that is bad. The does not appear to be truncated. It's over 600M, but the magic is wrong.

Digging into the file, it appears that the rpm header is placed incorrectly.

The signature header is in the correct place (offset 96), and appears to have a length of 4296. However the rpm header appears to start (judging by the magic) at offset 4400.

If this is happening consistently, that suggests that something is wrong with rpm on aarch64 in the buildroot.
Comment 6 Mattias Ellert 2016-12-17 18:58:42 EST
(In reply to Mike McLean from comment #4)
> If this is happening consistently, that suggests that something is wrong
> with rpm on aarch64 in the buildroot.

Thank you for your analysis. I reassign to rpm then.
Comment 7 Mattias Ellert 2016-12-18 04:28:27 EST
Now that it was established that it is the debuginfo package for aarch64 that is causing the problems, it was easier to know where to look for errors.

In the build.log for the build on aarch64 I see this error message repeated a few times during the execution of /usr/lib/rpm/find-debuginfo.sh:

readelf: Error: the dynamic segment offset + size exceeds the size of the file

It seems that rpm-build is ignoring this error and just goes on, resulting in it creating a broken rpm. This error does not appear in the logs for the other architectures.
Comment 8 Panu Matilainen 2017-01-04 10:54:11 EST
Right, this is actually not aarch64-specific at all but a generic rpmbuild regression originating from commit https://github.com/rpm-software-management/rpm/commit/68bddc353a7ea87ea00ad957858cd509e845e84c

The problem occurs when the estimated and actual payload size happen to be on different sides of UINT32_MAX which causes the signature header size tags to flip from 64bit to 32bit, causing the header to be off by eight bytes.

That this hasn't been seen on other architectures is just luck, but certainly arch-specific issues can make it more common (eg produced binaries being larger than on other arches)
Comment 10 Mattias Ellert 2017-01-23 08:23:43 EST
Will there be a fix for this before the F26 mass rebuild (scheduled 2017-02-01)?
Comment 11 Panu Matilainen 2017-01-23 08:28:05 EST
I've been intending to put out rpm 4.13.1 including this among other things ASAP but it seems somehow elusive... and certainly that's not going to happen this week due to Devconf. So I better fix it for the mass rebuild separately, thanks for the heads-up!
Comment 12 Panu Matilainen 2017-01-23 09:06:10 EST
Fixed in rawhide as of rpm-4.13.0-11.fc26.

As for other Fedora versions, the idea is to provide this and other fixes via upstream bugfix release.
Comment 13 Fedora Update System 2017-02-24 08:36:45 EST
rpm-4.13.0.1-1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-beb25f1953
Comment 14 Fedora Update System 2017-02-24 08:59:38 EST
rpm-4.13.0.1-1.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2017-54078f9dd2
Comment 15 Fedora Update System 2017-02-24 19:53:33 EST
rpm-4.13.0.1-1.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-54078f9dd2
Comment 16 Fedora Update System 2017-02-24 20:52:29 EST
rpm-4.13.0.1-1.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-beb25f1953
Comment 17 Fedora End Of Life 2017-02-28 05:48:35 EST
This bug appears to have been reported against 'rawhide' during the Fedora 26 development cycle.
Changing version to '26'.
Comment 18 Fedora Update System 2017-02-28 20:21:03 EST
rpm-4.13.0.1-1.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.
Comment 19 Fedora Update System 2017-02-28 20:25:02 EST
rpm-4.13.0.1-1.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.