Bug 1405570

Summary: koji build for rawhide fails with <class '_rpm.error'>: error reading package header
Product: [Fedora] Fedora Reporter: Mattias Ellert <mattias.ellert>
Component: rpmAssignee: Panu Matilainen <pmatilai>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 26CC: atigro, dennis, dominik, ignatenko, kardos.lubos, mhroncok, mikem, packaging-team-maint, pknirsch, pmatilai
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: aarch64   
OS: Unspecified   
URL: https://koji.fedoraproject.org/koji/taskinfo?taskID=16905568
Whiteboard:
Fixed In Version: rpm-4.13.0-11.fc26 rpm-4.13.0.1-1.fc24 rpm-4.13.0.1-1.fc25 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-03-01 01:21:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 245418, 1376324    

Description Mattias Ellert 2016-12-16 18:08:31 UTC
Description of problem:

I have now tried to build an update for Fedora Rawhide several times during the last week, and it fails consistently.

Build:
https://koji.fedoraproject.org/koji/buildinfo?buildID=823483

Tasks for the three attempts:
https://koji.fedoraproject.org/koji/taskinfo?taskID=16787974
https://koji.fedoraproject.org/koji/taskinfo?taskID=16796570
https://koji.fedoraproject.org/koji/taskinfo?taskID=16800637
https://koji.fedoraproject.org/koji/taskinfo?taskID=16856402
https://koji.fedoraproject.org/koji/taskinfo?taskID=16905568

For each of the attempts, the sub tasks of the srpm creation and the package builds for each architecture succeeds. There is however no tagBuild subtask started. When pressing the "Show results" link for the main task the same - to me very cryptic error message - is displayed for all of the attempts:

<class '_rpm.error'>: error reading package header

The same build for Fedora 25, Fedora 24 and EPEL 7 has competed without problems:

https://koji.fedoraproject.org/koji/buildinfo?buildID=823482
https://koji.fedoraproject.org/koji/buildinfo?buildID=823485
https://koji.fedoraproject.org/koji/buildinfo?buildID=823484

So the problem is Rawhide specific, and I can not tell what the cause is from the information I see en koji.

Version-Release number of selected component (if applicable):
The version on the Fedora Build system.

How reproducible:
Five times so far.

Steps to Reproduce:
1. See above

Actual results:
<class '_rpm.error'>: error reading package header

Expected results:
Successful build

Comment 1 Mike McLean 2016-12-16 19:18:33 UTC
I doubt this is a koji bug.

Looks like broken deps in the buildroot

DEBUG util.py:426:  nothing provides libpoppler.so.65()(64bit) needed by texlive-xetex-bin-6:svn41091-24.20160520.fc26.1.x86_64

Comment 2 Mattias Ellert 2016-12-16 20:10:03 UTC
You are looking a recent resubmit by kevin that suffers from a newly introduced problem that is unrelated to the reported bug. Please follow the links to the failed tasks in the bug report.

Comment 3 Mattias Ellert 2016-12-17 17:17:24 UTC
Texlive's broken deps due to the poppler soname bump was fixed by a rebuild.

Trying building again gives the same error as before:

https://koji.fedoraproject.org/koji/taskinfo?taskID=16921481

class '_rpm.error'>: error reading package header

Comment 4 Mike McLean 2016-12-17 19:31:18 UTC
One of the rpms is bad and does not read

# rpm -Kv /mnt/koji//work/tasks/1483/16921483/root-debuginfo-6.08.02-1.fc26.aarch64.rpm
error: /mnt/koji//work/tasks/1483/16921483/root-debuginfo-6.08.02-1.fc26.aarch64.rpm: headerRead failed: hdr magic: BAD

This is the only rpm in the set that is bad. The does not appear to be truncated. It's over 600M, but the magic is wrong.

Digging into the file, it appears that the rpm header is placed incorrectly.

The signature header is in the correct place (offset 96), and appears to have a length of 4296. However the rpm header appears to start (judging by the magic) at offset 4400.

If this is happening consistently, that suggests that something is wrong with rpm on aarch64 in the buildroot.

Comment 6 Mattias Ellert 2016-12-17 23:58:42 UTC
(In reply to Mike McLean from comment #4)
> If this is happening consistently, that suggests that something is wrong
> with rpm on aarch64 in the buildroot.

Thank you for your analysis. I reassign to rpm then.

Comment 7 Mattias Ellert 2016-12-18 09:28:27 UTC
Now that it was established that it is the debuginfo package for aarch64 that is causing the problems, it was easier to know where to look for errors.

In the build.log for the build on aarch64 I see this error message repeated a few times during the execution of /usr/lib/rpm/find-debuginfo.sh:

readelf: Error: the dynamic segment offset + size exceeds the size of the file

It seems that rpm-build is ignoring this error and just goes on, resulting in it creating a broken rpm. This error does not appear in the logs for the other architectures.

Comment 8 Panu Matilainen 2017-01-04 15:54:11 UTC
Right, this is actually not aarch64-specific at all but a generic rpmbuild regression originating from commit https://github.com/rpm-software-management/rpm/commit/68bddc353a7ea87ea00ad957858cd509e845e84c

The problem occurs when the estimated and actual payload size happen to be on different sides of UINT32_MAX which causes the signature header size tags to flip from 64bit to 32bit, causing the header to be off by eight bytes.

That this hasn't been seen on other architectures is just luck, but certainly arch-specific issues can make it more common (eg produced binaries being larger than on other arches)

Comment 10 Mattias Ellert 2017-01-23 13:23:43 UTC
Will there be a fix for this before the F26 mass rebuild (scheduled 2017-02-01)?

Comment 11 Panu Matilainen 2017-01-23 13:28:05 UTC
I've been intending to put out rpm 4.13.1 including this among other things ASAP but it seems somehow elusive... and certainly that's not going to happen this week due to Devconf. So I better fix it for the mass rebuild separately, thanks for the heads-up!

Comment 12 Panu Matilainen 2017-01-23 14:06:10 UTC
Fixed in rawhide as of rpm-4.13.0-11.fc26.

As for other Fedora versions, the idea is to provide this and other fixes via upstream bugfix release.

Comment 13 Fedora Update System 2017-02-24 13:36:45 UTC
rpm-4.13.0.1-1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-beb25f1953

Comment 14 Fedora Update System 2017-02-24 13:59:38 UTC
rpm-4.13.0.1-1.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2017-54078f9dd2

Comment 15 Fedora Update System 2017-02-25 00:53:33 UTC
rpm-4.13.0.1-1.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-54078f9dd2

Comment 16 Fedora Update System 2017-02-25 01:52:29 UTC
rpm-4.13.0.1-1.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-beb25f1953

Comment 17 Fedora End Of Life 2017-02-28 10:48:35 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 26 development cycle.
Changing version to '26'.

Comment 18 Fedora Update System 2017-03-01 01:21:03 UTC
rpm-4.13.0.1-1.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.

Comment 19 Fedora Update System 2017-03-01 01:25:02 UTC
rpm-4.13.0.1-1.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.