Why does this bug have the Needinfo flag set? What additional information do you need? Clearly rpm 4.8.x misuses read().

It's a needinfo on the developer (you are probably not authorized to see the recepient or I forgot to add the developer mail address)

Needinfo reset back on Panu to get his attention on this bug.

I suppose "oversight" is a fitting description, rpm2cpio being the only thing in rpm that accepts input from a pipe. I dont think I was even aware of it supporting reading from stdin prior to this bug...

While this technically is a regression, its on a very rarely used feature and the "normal" usage of 'rpm2cpio package.rpm|cpio ...' works just fine. The rpmio subsystem is such a house of cards that I'm not going to risk breaking more commonly used functionality with a hurried fix for a corner case issue  - moving to 6.4.0.

devel_ack, in 4.8.x this can be handled in the timedRead() wrapper easily enough. Upstream will need something different though...

I'm trying to reproduce this issue but no error occurred from 200 attempts.

# rpm -q rpm
rpm-4.8.0-27.el6.x86_64
# du -h large-1-1.noarch.rpm
190M	large-1-1.noarch.rpm
# for i in `seq 1 200`; do cat large-1-1.noarch.rpm | rpm2cpio 1>/dev/null; done
#

Any ideas? Is it possible that it depends on particular rpm used for test?

Maybe another scenario that can be used to verify the bug fix?

(In reply to comment #11)

> Any ideas? Is it possible that it depends on particular rpm used for test?

It will depend on OS and file system implementation. Maybe on memory pressure too. Try booting your system with mem=...

To reproduce you need a package which a header larger than system pipe buffer, the total size of the package (or amount of memory) is not important. Of course there tends to be a correlation: the packages with a large header tend to be large overall.

Anyway, the kernel package should be a fairly reliable reproducer.

Here's an actual reproducer (although you'll probably want to find a closer mirror for the kernel package):

$ wget http://ftp.funet.fi/pub/mirrors/fedora.redhat.com/pub/fedora/linux/releases/17/Fedora/x86_64/os/Packages/k/kernel-3.3.4-5.fc17.x86_64.rpm
$ cat kernel-3.3.4-5.fc17.x86_64.rpm |rpm2cpio|wc -l

Output with bug present will be something like:
error: rpm2cpio: headerRead failed: hdr blob(681753): BAD, read returned 129672
error reading header from package
0

Output with bug fixed (assuming same kernel package etc):
299593

Thanks Panu for the reproducer; it works well on my workstation but the success rate dramatically falls on any of the servers in our lab.

I checked the the pipe buffer size and it seems it is 65536 bytes on all system except ppc64 where it is 1048576 bytes. On ppc64 therefore I cannot reproduce it at all, but at least that makes sense. What I don't understand why it is that problem to reproduce it even on x86_64. Any idea? Is there anything else than the pipe buffer size that should be considered on the test systems?

I also tried to create a test package which contained nothing but a large changelog; that gain worked fine on my desktop but not on servers.

I dont see what else besides pipe buffer size could affect it. Where are you getting the pipe buffer size from?

Also... if everything else fails, use a bigger hammer ;) Bigger hammer being a larger package in this case: even the kernel header is below 1M in size, whereas header max size is 64M.

Do you need more than one reproducer? Isn't this a simple problem, fixed by correct use of read() inside fdRead():

http://linux.die.net/man/2/read

(In reply to comment #17)
> I dont see what else besides pipe buffer size could affect it. Where are you
> getting the pipe buffer size from?
> 
I used the script found here:
http://unix.stackexchange.com/questions/11946/how-big-is-the-pipe-buffer

> Also... if everything else fails, use a bigger hammer ;) Bigger hammer being
> a larger package in this case: even the kernel header is below 1M in size,
> whereas header max size is 64M.
Well, the big hammer seems to help. Having a spec file with 15M changelog gives quite good failure chances. (The poorest is on s390x, about 5%, but that is enough to verify the issue with higher attempts).

(In reply to comment #18)
> Do you need more than one reproducer? Isn't this a simple problem, fixed by
> correct use of read() inside fdRead():
> 
> http://linux.die.net/man/2/read
Yes, the problem is seems to be quite simple, but I need a deterministic way to reproduce the issue, so we can say there is not regression in the future.
Now I think we found the way.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0461.html