Bug 833427 - error: unpacking of archive failed on file ... cpio: Digest mismatch
error: unpacking of archive failed on file ... cpio: Digest mismatch
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: rpm (Show other bugs)
6.2
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: packaging-team-maint
Marek Marusic
:
: 1425417 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-06-19 08:59 EDT by Jonathan Underwood
Modified: 2017-02-22 07:18 EST (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Although the RPM Package Manager does not support packages with files larger than 4 GB, the rpm utility allowed creating source packages where individual files exceeded 4 GB. The installation of such packages then failed with a "Digest mismatch" error. Now, rpm no longer allows the creation of such packages, which in turn prevents the described installation failure.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-07-22 03:01:15 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Jonathan Underwood 2012-06-19 08:59:45 EDT
Description of problem:
I have a src rpm I created a while ago on a Redhat 6.0 system which fails to install on a Redhat 6.2 system:

$ rpm -ivh matlab-2010b-3.el6.src.rpm 
   1:matlab                 ########################################### [100%]
error: unpacking of archive failed on file /local/home/rpmb/rpmbuild/SOURCES/matlab-2010b.tar.gz;4fe075b0: cpio: Digest mismatch
error: matlab-2010b-3.el6.src.rpm cannot be installed


Version-Release number of selected component (if applicable):
$ rpm -qa | grep rpm
rpm-build-4.8.0-19.el6_2.1.x86_64
python-deltarpm-3.5-0.5.20090913git.el6.x86_64
rpmdevtools-7.5-1.el6.noarch
rpm-libs-4.8.0-19.el6_2.1.x86_64
redhat-rpm-config-9.0.3-34.el6.sl.noarch
deltarpm-3.5-0.5.20090913git.el6.x86_64
eclipse-rpm-editor-0.5.0-2.el6.x86_64
rpm-4.8.0-19.el6_2.1.x86_64
rpmlint-0.94-2.el6.noarch
rpm-python-4.8.0-19.el6_2.1.x86_64
Comment 2 Jonathan Underwood 2012-06-19 10:08:54 EDT
Changing component to cpio.

$ rpm -q cpio
cpio-2.10-10.el6.x86_64
Comment 3 Jonathan Underwood 2012-06-19 10:09:26 EDT
$ rpm2cpio matlab-2010b-3.el6.src.rpm | cpio -t
matlab-2010b.tar.gz
cpio: premature end of file
Comment 4 Jonathan Underwood 2012-06-19 13:23:44 EDT
In fact creating a SRC RPM on RHEL 6.2, and then subsequently trying to install that src rpm breaks in the same way. This worked on RHEL 6.0. Of note here is that the SRC RPM is 4.8 GB big.

This is a pretty bad regression.
Comment 5 Ondrej Vasik 2012-06-20 00:03:36 EDT
Thanks for your report. Afaik rpm uses its own implementation of cpio, so it is possible that it was broken for large files at the time of the RHEL-6.0 - and unpacking it directly via cpio just fails - as the file is probably incomplete or incompatible with original cpio format. Is there any reasoning why you moved this bugzilla from rpm to cpio? Just based on cpio complains about premature end of file? Have you tried using RHEL-6.0 cpio for unpacking?
Comment 6 Panu Matilainen 2012-06-20 03:15:39 EDT
I've a hard time seeing what change in rpm might have caused such a thing, but cpio bug this is not as it's never involved in creation or extraction of packages -> switching back to rpm component.

Since a previously working src.rpm fails to install (well, extract) on rhel-6.2 it'd seem that the payload gets truncated on extract phase (as opposed to creation), for whatever reason. Can you easily test whether the src.rpm created on rhel-6.2 is installable on 6.0?
Comment 7 Jonathan Underwood 2012-06-20 12:13:16 EDT
(In reply to comment #6)
> I've a hard time seeing what change in rpm might have caused such a thing,
> but cpio bug this is not as it's never involved in creation or extraction of
> packages -> switching back to rpm component.
> 

OK, sorry, I naively assumed that rpm used cpio to create and extract rpms. Nonetheless, there is something that has changed to break *both* cpio and rpm.

> Since a previously working src.rpm fails to install (well, extract) on
> rhel-6.2 it'd seem that the payload gets truncated on extract phase (as
> opposed to creation), for whatever reason. Can you easily test whether the
> src.rpm created on rhel-6.2 is installable on 6.0?

I can't check that as I no longer have any 6.0 machines alas. 

But, what I can tell you is:

1) The RPM created under RHEL 6.0 cannot be installed by rpm from RHEL 6.2

2) The RPM created under RHEL 6.0 cannot be unpacked using cpio from RHEL 6.2

3) An RPM created using rpm from RHEL 6.2 cannot be installed using the same rpm binary from RHEL 6.2

4) An RPM created using rpm from RHEL 6.2 cannot be unpacked using cpio from RHEL 6.2

This is for an rpm containing a large (4.8 GB) tar.gz file.

So, I don't think it's particularly something to do with a RHEL 6.0 - 6.2 compatibility, as things are broking consistently within 6.2.

Let me know if thwere's anything else I can do to debug.
Comment 8 Panu Matilainen 2012-06-21 04:58:58 EDT
4.8GB tarball source? Ouch. That's actually larger than rpm can currently support, the limit for single file in a package is UINT32_MAX bytes (due to the underlying cpio format limitation). This is checked for files in binary packages but source packages get generated a bit differently and that check, along with various other "large package" aspects is missing for src.rpm's :-/

So the actual bug here is that rpm should've never allowed packaging that thing in the first place, and failure to install/extract is in fact "expected." The uint32_t type used for file sizes gets truncated or wraps around at build-time, causing install/extract to see a shorter file than it is and things go downhill from there.

I fail to see how the src.rpm could've been successfully installed on rhel-6.0. Perhaps you never actually installed the src.rpm on rhel-6.0 either, only (re)built from a spec?

There are (at least) two things needing fixing here, devel_ack for these:
1) Add checks for the maximum allowed file size for sources and patches in src.rpm
2) Fix (or rather, add) "large package" support for src.rpm's as well similar to binary packages (ie while individual file limit is 4GB, the total archive can be much larger but it needs special tracking which is currently missing for src.rpm)

Neither of which is going to directly fix your issue (the cpio format limit will almost certainly not be fixed in rhel-6 or even -7), but its possible to work around it by splitting the sources into max 4GB pieces. That will probably "work" even now, although there will be anomalies with reported total archive sizes and such.
Comment 9 Jonathan Underwood 2012-06-21 19:15:31 EDT
Hm. Right, thanks for taking the time to explain that. Makes sense.

It would be nice to see those enhancements land in RPM at some point though - the use case here is deploying a proprietory program (Matlab) as an RPM by simply bundling the whole thing up, and running the matlab installer in %install. Being able to deply binary software like this as an rpm is a great help when deploying many machines using tooling that makes use of yum repos (katello, pulp, foreman etc).

Back when I was on RHEL 6.0 I was building the SRPM and then successfully generating binary rpms from that SRPM using mock. So it was definitely unpacking OK. However, I will go back and check the file size carefully. The newest matlab is 4.8 GB, perhaps the old one was less than 4.0 GB. Still, I wonder why it won't unpack on 6.2, when it seemed to be doing so on 6.0. Anyway, will check more carefully and see if i can make sense of it.

Thanks again for taking the time to explain the issues. I may even see if I can work up a patch for rpm to at least have it refuse to build broken srpms :)
Comment 10 RHEL Product and Program Management 2012-07-10 04:10:15 EDT
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.
Comment 11 RHEL Product and Program Management 2012-07-10 21:52:18 EDT
This request was erroneously removed from consideration in Red Hat Enterprise Linux 6.4, which is currently under development.  This request will be evaluated for inclusion in Red Hat Enterprise Linux 6.4.
Comment 12 RHEL Product and Program Management 2012-08-14 17:59:52 EDT
This request was evaluated by Red Hat Product Management for
inclusion in a Red Hat Enterprise Linux release.  Product
Management has requested further review of this request by
Red Hat Engineering, for potential inclusion in a Red Hat
Enterprise Linux release for currently deployed products.
This request is not yet committed for inclusion in a release.
Comment 14 RHEL Product and Program Management 2012-12-14 03:12:51 EST
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.
Comment 16 RHEL Product and Program Management 2013-07-25 17:11:09 EDT
This request was evaluated by Red Hat Product Management for
inclusion in a Red Hat Enterprise Linux release.  Product
Management has requested further review of this request by
Red Hat Engineering, for potential inclusion in a Red Hat
Enterprise Linux release for currently deployed products.
This request is not yet committed for inclusion in a release.
Comment 17 RHEL Product and Program Management 2013-10-14 00:54:40 EDT
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.
Comment 19 Florian Festi 2014-12-15 07:48:41 EST
Support for files with > 4GB was added upstream but the changes are much too massive to port them back to an already released RHEL version.

We still should issue an error so such broken packages cannot be build in the first place. So devel-ack purely for adding this check.
Comment 24 errata-xmlrpc 2015-07-22 03:01:15 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-1452.html
Comment 25 abdul 2017-02-21 05:59:28 EST
I Could someone help here to fix the same kind of issue related to Xilinx-SDK

[root@euca-10-254-169-180 SRPMS]# rpm -ivh xilinx-sdk_v20164-2016.4-1.src.rpm
   1: xilinx-sdk_v2016########################################### [100%]
error: unpacking of archive failed on file /root/rpmbuild/SOURCES/xilinx-sdk.tar.gz;58ac1c7f: cpio: Digest mismatch
error: xilinx-sdk_v20164-2016.4-1.src.rpm cannot be installed
[root@euca-10-254-169-180 SRPMS]#

Here the size of the source file is of 14GB and any idea how i can have a workaround for this issue ?
Comment 26 Panu Matilainen 2017-02-22 00:26:17 EST
*** Bug 1425417 has been marked as a duplicate of this bug. ***
Comment 27 Panu Matilainen 2017-02-22 00:30:33 EST
The workaround is to split the source tarball up into < 4GB chunks and catenate them back in %prep before unpacking. Whether it's really worth the trouble is another question, it's rather desperate.
Comment 28 Panu Matilainen 2017-02-22 00:37:26 EST
Oh and for the record, support for files larger than 4GB in the payload was added in rpm 4.12 so it's not in RHEL 7 either, but current Fedora versions do.
Comment 29 abdul 2017-02-22 04:49:39 EST
Hi Guys,

Thanks for the input here ,
May i know how i can have the %prep section if my source files are splitted into 5 sub-tar balls.
Source6: xilinx-sdk-1.tar.gz
Source7: xilinx-sdk-2.tar.gz
Source8: xilinx-sdk-3.tar.gz
Source9: xilinx-sdk-4.tar.gz
Source10: xilinx-sdk-5.tar.gz

%prep
??
??

Any idea ? Can someone help here pls.
Comment 30 Pavel Raiskup 2017-02-22 06:37:57 EST
%prep
cat xilinx-sdk-*.tar.gz > %SOURCE0 # if 'Source0:  xilinx-sdk.tar.gz'
%setup ...

Btw:
$ split --verbose -b ... xilinx-sdk.tar.gz xilinx-sdk.tar.gz.'
$ truncate -s 0 xilinx-sdk.tar.gz

Would allow you to do
%prep
cat %SOURCE0.* > %SOURCE0
Comment 31 abdul 2017-02-22 07:18:01 EST
Do we have to add the whole commands you printed here or just we need to add the :
prep
cat xilinx-sdk-*.tar.gz > %SOURCE0 # if 'Source0:  xilinx-sdk.tar.gz'
%setup ...

1) query is that , how could we use the %setup command  without any options.

Would you be able to provide little more clarity on this pls.

Note You need to log in before you can comment on or make changes to this bug.