We have an /etc/group file with several very long lines (the longest is 1747 chars). I built a qmail rpm which has files in it that belong to the groups "nofiles" or "qmail". When I have those two groups in our /etc/group file *before* the long lines there is no problem. When I put those groups in /etc/group *after* the long lines rpm prints group qmail does not exist - using root when installing this rpm. I am using rpm-3.0.2-5.x both to build and install the rpm.
This isn't an rpm problem per se, since rpm does not access /etc/group directly, but only uses getgrnam/getgrgid interfaces from glibc. (Note: there is a hard coded maximum 1024 group limit in rpm, but I don't believe that you are reaching that limit). Off to glibc for resolution ...
Still present in 8.0
I cannot reproduce this with RHL9. I have at the beginning of /etc/group a line with >2000 characters. My user entry is behind this and rpmbuild happily creates an output file even though the files which get added belong to me. Please check this out yourself and report back.