Created attachment 412973 [details]
log of rpmbuild

(In reply to comment #0)
correct cut-n-paste error

> # NFS mounted rpmbuild dirs:
> TOPDIR= /auto/ws1/rjohnson/ftl-8.0/trunk/ftl_modified/scratch
> rpmbuild -ba ${TOPDIR}/device-mapper-multipath.spec \
>     --buildroot=${TOPDIR}/InstallStaging --define="_topdir ${TOPDIR}"

should be:

TOPDIR= /auto/ws1/rjohnson/ftl-8.0/trunk/ftl_modified/scratch
 rpmbuild -ba ${TOPDIR}/SPECS/device-mapper-multipath.spec \
     --buildroot=${TOPDIR}/InstallStaging --define="_topdir ${TOPDIR}"

Setting the buildroot on a NFS mount encourages problems (is more fragile) (timeskew also rears its head occasionally and confuses 'make')

Adding 'lockfile/dotfile' whiteout is presently supported with a -f file manifest, to the taste of the end spec file writer

Using a local spindle avoids this altogether

The old addage: 'dont fight city hall' by altering your build practices by confirming to one or more of the above, comes to mind to solve your problem tactically

-- Russ herrold

This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.

Stratus has used NFS to access a working copy during rpmbuild since RHEL4.  Naively, this seems to be a regression introduced sometime after RHEL6 Alpha3.

no argument ... just saying that there is a workaround readily available

best regards

- Russ herrold

(In reply to comment #4)

> Adding 'lockfile/dotfile' whiteout is presently supported with a -f file
> manifest, to the taste of the end spec file writer
> 
What would be the syntax to specify the whiteout? 
Would you have an example, or reference to an example or appnote?

All I can find regarding
 %files -f <manifest>
indicates that <manifest>:
  - can be created dynamically, presumably by %prep, %build or %install
  - contains directives otherwise present in the %files section
  - operates in union with directives present in the %files 
My search has turned up nothing regarding whiteout lists.

The term 'whiteout' is used in some complex RPM packagings to indicate the process of making certain files 'not appear' when they would otherwise be found by a match pattern.

Looking in my 'spec file archive, I find the -f option used a lot

[herrold@centos-5 SPECS]$ grep "%files -f " *.spec | wc
    155     476    6609
of 841 total spec files

Looking at one chosen at random:
   perl-Digest-HMAC.spec
which has been in Red Hat for a long time (at least since 2002), we see:

find $RPM_BUILD_ROOT \( -name perllocal.pod -o -name .packlist \) -exec rm -v {}
 \;

find $RPM_BUILD_ROOT/usr -type f -print | \
        sed "s@^$RPM_BUILD_ROOT@@g" | \
        grep -v perllocal.pod | \
        grep -v "\.packlist" > Digest-HMAC-%{version}-filelist
if [ "$(cat Digest-HMAC-%{version}-filelist)X" = "X" ] ; then
    echo "ERROR: EMPTY FILE LIST"
    exit -1
fi

%files -f Digest-HMAC-%{version}-filelist
%defattr(-,root,root)

%changelog
    ...
===================================================================

That is, the first find argument uses -exec to affirmatively remove files that are of a general nature and rebuilt during an install.

The SECOND, however, has two "grep -v" in the pipeline producing the -f list which seemingly do the same function but only down the '$RPM_BUILD_ROOT/usr' part of the installation root path

That is, it applies 'whiteout' to remove unwanted files from that file manifest

In your case, it seems an eliding match on:  
    | grep -v '[/][.]nfs[0-9]+'  
would work

-- Russ herrold

NFS has always produced these weird rename files. I think that you will have no alternative but to fix this in the rpmbuild.

What kernel is the client running here?

Symptoms appeared with client running RHEL6.0-snapshot1:  2.6.32-22.el6.x86_64
Symptoms were absent with client running RHEL6.0-Beta1:  2.6.32-19.el6.x86_64

Thanks, it's probably related to the NFS writeback changes that went in after -19.el6. Just to make sure, I assume that your builds don't involve anything strange like multiple clients deleting and moving files around in the directory at the same time?

If not, then we'll move this to a kernel bug and see whether some of the patches I have queued up for some other problems may fix this.

FWIW, the only changes to rpm between Beta1 and Snapshot1 are related to package installation from remote url and verifying installed packages, and neither of the fixes touch code paths that get executed on package building. Reassigning as Jeff apparently already has a prime suspect in the kernel land.

(In reply to comment #15)
> Thanks, it's probably related to the NFS writeback changes that went in after
> -19.el6. Just to make sure, I assume that your builds don't involve anything
> strange like multiple clients deleting and moving files around in the directory
> at the same time?

Correct.  

FWIW, the symptoms can be reproduced by rebuilding RH distributed SRPMS with an NFS mounted _topdir  For example:
  TOPDIR=<some-NFS-dir>
  mount -oloop,ro .../RHEL6.0-20100428.3-SRPMS-DVD2.iso /mnt
  rpmbuild --rebuild --define="_topdir ${TOPDIR}" /mnt/SRPMS/device-mapper-multipath-0.4.9-18.el6.src.rpm

Hey Rich,

This problem is fixed in kernel-2.6.32-24.el6 which I believe you
should be seeing by the end of the today... If this is not
the case, let me know and I will make the appropriate arrangements
to get that kernel...

Thanks,  I'll try it out as soon as I can.

Stratus has observed this problem no longer occurs once our build machines (NFS clients) upgraded to RHEL6 Beta1 snapshot 4 (kernel-2.6.32-25.el6)