Bug 590886 - [Stratus 6.0 bug] rpmbuild fails when buildroot is NFS mounted.
[Stratus 6.0 bug] rpmbuild fails when buildroot is NFS mounted.
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kernel (Show other bugs)
6.0
All Linux
medium Severity medium
: rc
: 6.0
Assigned To: Steve Dickson
Red Hat Kernel QE team
: OtherQA, Regression
Depends On:
Blocks: 554559
  Show dependency treegraph
 
Reported: 2010-05-10 17:20 EDT by Rich Johnson
Modified: 2010-05-25 17:20 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-05-25 17:20:49 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
log of rpmbuild (29.55 KB, text/plain)
2010-05-10 17:21 EDT, Rich Johnson
no flags Details

  None (edit)
Description Rich Johnson 2010-05-10 17:20:14 EDT
Description of problem:
 /usr/lib/rpm/check-files objects to .nfsXXX temporary files created
during /usr/lib/rpm/find-debuginfo.sh 

Version-Release number of selected component (if applicable):
 - rpm-4.8.0-5.el6.x86_64


How reproducible:
  Invoke rpmbuild with  "--buildroot=<some-nfs-dir>" and/or " --define='_topdir <some-nfs-dir>'"

  For example, using the NFS auto mounted directory "auto/ws1/rjohnson/ftl-8.0/trunk/ftl_modified/scratch" containing an rpmbuild hierachy of SPECS, SOURCES, BUILD, etc.

# NFS mounted rpmbuild dirs:
TOPDIR= /auto/ws1/rjohnson/ftl-8.0/trunk/ftl_modified/scratch
rpmbuild -ba ${TOPDIR}/device-mapper-multipath.spec \
    --buildroot=${TOPDIR}/InstallStaging --define="_topdir ${TOPDIR}"
  
Actual results:

[ excerpt from rpmbuild output; the full log is attached ]

[...SNIP...]
Processing files: device-mapper-multipath-libs-0.4.9-4.el6_ftSys_8.0.0_53.x86_64
warning: File listed twice: /lib64/multipath
Provides: .nfs00000000049640c50000052b()(64bit) .nfs00000000049640c60000052c()(64bit) .nfs00000000049641020000052d()(64bit) .nfs00000000049641030000052e()(64bit) .nfs00000000049641040000052f()(64bit) .nfs000000000496410500000530()(64bit) .nfs000000000496410600000531()(64bit) .nfs000000000496411400000532()(64bit) .nfs000000000496411700000533()(64bit) .nfs000000000496411900000534()(64bit) .nfs000000000496411a00000535()(64bit) .nfs000000000496411b00000536()(64bit) .nfs000000000496411c00000537()(64bit) .nfs000000000496411d00000538()(64bit) .nfs000000000496411e00000539()(64bit) device-mapper-multipath-libs = 0.4.9-4.el6_ftSys_8.0.0_53 libcheckcciss_tur.so()(64bit) libcheckdirectio.so()(64bit) libcheckemc_clariion.so()(64bit) libcheckhp_sw.so()(64bit) libcheckrdac.so()(64bit) libcheckreadsector0.so()(64bit) libchecktur.so()(64bit) libmultipath.so()(64bit) libprioalua.so()(64bit) libprioconst.so()(64bit) libprioemc.so()(64bit) libpriohds.so()(64bit) libpriohp_sw.so()(64bit) libprionetapp.so()(64bit) libpriorandom.so()(64bit) libpriordac.so()(64bit)
Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(VersionedDependencies) <= 3.0.3-1
Requires: libaio.so.1()(64bit) libaio.so.1(LIBAIO_0.1)(64bit) libaio.so.1(LIBAIO_0.4)(64bit) libc.so.6()(64bit) libc.so.6(GLIBC_2.2.5)(64bit) libc.so.6(GLIBC_2.3)(64bit) libc.so.6(GLIBC_2.3.2)(64bit) rtld(GNU_HASH)
Processing files: kpartx-0.4.9-4.el6_ftSys_8.0.0_53.x86_64
Provides: kpartx = 0.4.9-4.el6_ftSys_8.0.0_53
Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(VersionedDependencies) <= 3.0.3-1
Requires: libc.so.6()(64bit) libc.so.6(GLIBC_2.2.5)(64bit) libc.so.6(GLIBC_2.3)(64bit) libc.so.6(GLIBC_2.3.3)(64bit) libdevmapper.so.1.02()(64bit) libdevmapper.so.1.02(Base)(64bit) rtld(GNU_HASH)
Processing files: device-mapper-multipath-debuginfo-0.4.9-4.el6_ftSys_8.0.0_53.x86_64
Checking for unpackaged file(s): /usr/lib/rpm/check-files /auto/ws1/rjohnson/ftl-8.0/trunk/ftl_modified/scratch/InstallStaging
error: Installed (but unpackaged) file(s) found:
   /lib64/.nfs000000000495c1fe0000052a
   /sbin/.nfs00000000017d415e0000053a
   /sbin/.nfs00000000017d41610000053b
   /sbin/.nfs00000000017d41630000053c
   /sbin/.nfs00000000017d41650000053d


RPM build errors:
    File listed twice: /lib64/multipath
    Installed (but unpackaged) file(s) found:
   /lib64/.nfs000000000495c1fe0000052a
   /sbin/.nfs00000000017d415e0000053a
   /sbin/.nfs00000000017d41610000053b
   /sbin/.nfs00000000017d41630000053c
   /sbin/.nfs00000000017d41650000053d
-bash-4.1$ 


Expected results:
  - successful rpmbuild
  - .nfsXXXX temporary files are an implementation detail of the NFS server and should not affect client behavior.   As such they should be either:
     - not exposed to the client, or
     - filtered/ignored by the client.

Additional info:
- NFS server is running redhat-release-4AS-9
Comment 1 Rich Johnson 2010-05-10 17:21:55 EDT
Created attachment 412973 [details]
log of rpmbuild
Comment 2 Rich Johnson 2010-05-10 17:26:33 EDT
(In reply to comment #0)
correct cut-n-paste error

> # NFS mounted rpmbuild dirs:
> TOPDIR= /auto/ws1/rjohnson/ftl-8.0/trunk/ftl_modified/scratch
> rpmbuild -ba ${TOPDIR}/device-mapper-multipath.spec \
>     --buildroot=${TOPDIR}/InstallStaging --define="_topdir ${TOPDIR}"

should be:

TOPDIR= /auto/ws1/rjohnson/ftl-8.0/trunk/ftl_modified/scratch
 rpmbuild -ba ${TOPDIR}/SPECS/device-mapper-multipath.spec \
     --buildroot=${TOPDIR}/InstallStaging --define="_topdir ${TOPDIR}"
Comment 4 R P Herrold 2010-05-10 18:48:43 EDT
Setting the buildroot on a NFS mount encourages problems (is more fragile) (timeskew also rears its head occasionally and confuses 'make')

Adding 'lockfile/dotfile' whiteout is presently supported with a -f file manifest, to the taste of the end spec file writer

Using a local spindle avoids this altogether

The old addage: 'dont fight city hall' by altering your build practices by confirming to one or more of the above, comes to mind to solve your problem tactically

-- Russ herrold
Comment 5 RHEL Product and Program Management 2010-05-10 18:51:08 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.
Comment 6 Robert N. Evans 2010-05-10 19:00:49 EDT
Stratus has used NFS to access a working copy during rpmbuild since RHEL4.  Naively, this seems to be a regression introduced sometime after RHEL6 Alpha3.
Comment 7 R P Herrold 2010-05-10 19:12:41 EDT
no argument ... just saying that there is a workaround readily available

best regards

- Russ herrold
Comment 8 Rich Johnson 2010-05-11 09:10:21 EDT
(In reply to comment #4)

> Adding 'lockfile/dotfile' whiteout is presently supported with a -f file
> manifest, to the taste of the end spec file writer
> 
What would be the syntax to specify the whiteout? 
Would you have an example, or reference to an example or appnote?

All I can find regarding
 %files -f <manifest>
indicates that <manifest>:
  - can be created dynamically, presumably by %prep, %build or %install
  - contains directives otherwise present in the %files section
  - operates in union with directives present in the %files 
My search has turned up nothing regarding whiteout lists.
Comment 9 R P Herrold 2010-05-11 12:15:15 EDT
The term 'whiteout' is used in some complex RPM packagings to indicate the process of making certain files 'not appear' when they would otherwise be found by a match pattern.

Looking in my 'spec file archive, I find the -f option used a lot

[herrold@centos-5 SPECS]$ grep "%files -f " *.spec | wc
    155     476    6609
of 841 total spec files

Looking at one chosen at random:
   perl-Digest-HMAC.spec
which has been in Red Hat for a long time (at least since 2002), we see:

find $RPM_BUILD_ROOT \( -name perllocal.pod -o -name .packlist \) -exec rm -v {}
 \;

find $RPM_BUILD_ROOT/usr -type f -print | \
        sed "s@^$RPM_BUILD_ROOT@@g" | \
        grep -v perllocal.pod | \
        grep -v "\.packlist" > Digest-HMAC-%{version}-filelist
if [ "$(cat Digest-HMAC-%{version}-filelist)X" = "X" ] ; then
    echo "ERROR: EMPTY FILE LIST"
    exit -1
fi

%files -f Digest-HMAC-%{version}-filelist
%defattr(-,root,root)

%changelog
    ...
===================================================================

That is, the first find argument uses -exec to affirmatively remove files that are of a general nature and rebuilt during an install.

The SECOND, however, has two "grep -v" in the pipeline producing the -f list which seemingly do the same function but only down the '$RPM_BUILD_ROOT/usr' part of the installation root path

That is, it applies 'whiteout' to remove unwanted files from that file manifest

In your case, it seems an eliding match on:  
    | grep -v '[/][.]nfs[0-9]+'  
would work

-- Russ herrold
Comment 11 Ric Wheeler 2010-05-11 14:17:11 EDT
NFS has always produced these weird rename files. I think that you will have no alternative but to fix this in the rpmbuild.
Comment 12 Jeff Layton 2010-05-11 14:34:52 EDT
What kernel is the client running here?
Comment 13 Rich Johnson 2010-05-11 15:19:54 EDT
Symptoms appeared with client running RHEL6.0-snapshot1:  2.6.32-22.el6.x86_64
Symptoms were absent with client running RHEL6.0-Beta1:  2.6.32-19.el6.x86_64
Comment 15 Jeff Layton 2010-05-11 15:37:41 EDT
Thanks, it's probably related to the NFS writeback changes that went in after -19.el6. Just to make sure, I assume that your builds don't involve anything strange like multiple clients deleting and moving files around in the directory at the same time?

If not, then we'll move this to a kernel bug and see whether some of the patches I have queued up for some other problems may fix this.
Comment 17 Panu Matilainen 2010-05-12 01:55:10 EDT
FWIW, the only changes to rpm between Beta1 and Snapshot1 are related to package installation from remote url and verifying installed packages, and neither of the fixes touch code paths that get executed on package building. Reassigning as Jeff apparently already has a prime suspect in the kernel land.
Comment 18 Rich Johnson 2010-05-12 06:43:57 EDT
(In reply to comment #15)
> Thanks, it's probably related to the NFS writeback changes that went in after
> -19.el6. Just to make sure, I assume that your builds don't involve anything
> strange like multiple clients deleting and moving files around in the directory
> at the same time?

Correct.  

FWIW, the symptoms can be reproduced by rebuilding RH distributed SRPMS with an NFS mounted _topdir  For example:
  TOPDIR=<some-NFS-dir>
  mount -oloop,ro .../RHEL6.0-20100428.3-SRPMS-DVD2.iso /mnt
  rpmbuild --rebuild --define="_topdir ${TOPDIR}" /mnt/SRPMS/device-mapper-multipath-0.4.9-18.el6.src.rpm
Comment 22 Steve Dickson 2010-05-12 11:24:45 EDT
Hey Rich,

This problem is fixed in kernel-2.6.32-24.el6 which I believe you
should be seeing by the end of the today... If this is not
the case, let me know and I will make the appropriate arrangements
to get that kernel...
Comment 24 Rich Johnson 2010-05-12 11:57:13 EDT
Thanks,  I'll try it out as soon as I can.
Comment 25 Robert N. Evans 2010-05-25 14:25:53 EDT
Stratus has observed this problem no longer occurs once our build machines (NFS clients) upgraded to RHEL6 Beta1 snapshot 4 (kernel-2.6.32-25.el6)

Note You need to log in before you can comment on or make changes to this bug.