Bug 1073004 - Package fails to rebuild on Rawhide with test suite failures
Summary: Package fails to rebuild on Rawhide with test suite failures
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Fedora
Classification: Fedora
Component: apache-commons-io
Version: rawhide
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
Assignee: Mikolaj Izdebski
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-03-05 15:27 UTC by Alexander Todorov
Modified: 2014-03-07 14:55 UTC (History)
5 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2014-03-07 11:39:30 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
build log with debug enabled (998.24 KB, text/plain)
2014-03-07 09:37 UTC, Alexander Todorov
no flags Details
surefire-reports.tar.gz (23.99 KB, application/x-gzip)
2014-03-07 09:38 UTC, Alexander Todorov
no flags Details

Description Alexander Todorov 2014-03-05 15:27:28 UTC
Description of problem:
Test suite failures prevent the package from rebuilding on Rawhide (updated today)

Version-Release number of selected component (if applicable):
apache-commons-io-2.4-11.fc21
maven-3.2.1-3.fc21.noarch


How reproducible:
Always (not sure)

Steps to Reproduce:
1. Download pache-commons-io-2.4-11.fc21 from Koji
2. rpmbuild --rebuild pache-commons-io-2.4-11.fc21.src.rpm
3.

Actual results:


Results :

When rebuilding as regular user:

Tests in error: 
  FileUtilsTestCase.testCopyDirectoryToNonExistingDest:1172 ? IO Unable to delet...
  FileUtilsTestCase.testCopyDirectoryToDirectory_NonExistingDest:1149 ? IO Unabl...
  FileUtilsTestCase.testCopyDirectoryToExistingDest:1193 ? IO Unable to delete f...

Tests run: 966, Failures: 0, Errors: 3, Skipped: 0



When rebuilding as root:

Results :

Failed tests: 
  FileFilterTestCase.testCanRead:847->assertFiltering:65 Filter(File) org.apache.commons.io.filefilter.AndFileFilter not true for /root/rpmbuild/BUILD/commons-io-2.4-src/test/io/read-only-file1.txt expected:<true> but was:<false>
  FileFilterTestCase.testCanWrite:857->assertFiltering:65 Filter(File) org.apache.commons.io.filefilter.CanWriteFileFilter not false for /root/rpmbuild/BUILD/commons-io-2.4-src/test/io/read-only-file2.txt expected:<false> but was:<true>
  FileUtilsCleanDirectoryTestCase.testThrowsOnNullList:102 expected IOException
  FileUtilsCleanDirectoryTestCase.testThrowsOnCannotDeleteFile:121 expected IOException

Tests run: 966, Failures: 4, Errors: 0, Skipped: 0

Comment 1 Stanislav Ochotnicky 2014-03-05 16:29:10 UTC
I can't reproduce at all
 * fedpkg local on F20 works (Tests run: 966, Failures: 0, Errors: 0, Skipped: 0)
 * scratchbuild in rawhide works: http://koji.fedoraproject.org/koji/taskinfo?taskID=6600787

I'd say this is possibly a PEBKAC, SELinux issue in rawhide or a weird coincidence. Closing as notabug, but maybe if you can reliably reproduce in a clean rawhide VM it might make sense to track down reason.

Comment 2 Alexander Todorov 2014-03-06 08:53:45 UTC
No SELinux errors as far as I can see. What is PEBKAC ? 

Trying to install Rawhide in a clean VM (original test system is F20 upgraded to Rawhide) and reproduce.

Comment 3 Alexander Todorov 2014-03-06 10:02:34 UTC
I can reproduce reliably.

1) Prepare a fresh VM using ISO from http://koji.fedoraproject.org/koji/taskinfo?taskID=6602517 (latest I could find)
2) Reboot into the new system
3) Enable rawhide-source repo
4) yum clean all && yum -y update (didn't find any more updates)
5) yum -y install yum-utils maven rpm-build maven-local apache-commons-parent
6) setenforce 0 (although I didn't see any SELinux related errors)

7) yumdownloader --source apache-commons-io
8) rpmbuild --rebuild apache-commons-io-2.4-11.fc21.src.rpm 


Build fails.

Comment 4 Mikolaj Izdebski 2014-03-06 10:34:14 UTC
Not reproducible for me either.  All tests pass and the build completes successfully.

Comment 5 Alexander Todorov 2014-03-06 15:08:54 UTC
(In reply to Mikolaj Izdebski from comment #4)
> Not reproducible for me either.  All tests pass and the build completes
> successfully.

How did you try to reproduce? I managed to hit this error several times already. 

Did you follow comment #3 ?

Comment 6 Alexander Todorov 2014-03-06 15:26:52 UTC
NB: I can reproduce this with both

apache-commons-io-2.4-11.fc21.src.rpm on Rawhide or
apache-commons-io-2.4-10.fc20.src.rpm on Fedora 20

both systems are fresh install updated to the latest available updates. 


Describe your steps to reproduce in order to narrow down the root cause of this.

Comment 7 Mikolaj Izdebski 2014-03-06 15:31:39 UTC
(In reply to Alexander Todorov from comment #5)
> (In reply to Mikolaj Izdebski from comment #4)
> > Not reproducible for me either.  All tests pass and the build completes
> > successfully.
> 
> How did you try to reproduce? I managed to hit this error several times
> already. 

I've built the package in F21 chroot.

> Did you follow comment #3 ?

No, I don't think I ever used Fedora live CD.

Isn't LiceCD using some unionfs/aufs to merge contents of tmpfs and iso9660? That could be a file system limitation.

If you want this to be resolved you'll have to provide extra information:

1) file system type and mount flags used (attach appriopriate line from /etc/mtab)

2) full build logs with debugging output enabled (append -X to line containing %mvn_build)

3) surefire test logs, available under target/surefire-reports/ after build fails

Comment 8 Alexander Todorov 2014-03-06 15:34:12 UTC
FYI the files
/root/rpmbuild/BUILD/commons-io-2.4-src/test/io/read-only-file1.txt
/root/rpmbuild/BUILD/commons-io-2.4-src/test/io/read-only-file2.txt

do not exist on my system.

Comment 9 Mikolaj Izdebski 2014-03-06 15:37:50 UTC
My steps to reproduce: run Koji build:
koji build --scratch --nowait rawhide "git://pkgs.fedoraproject.org/apache-commons-io?#HEAD"

Alternatively: install @buildsys-build into chroot using YUM, run yum-builddep to install build dependencies of apache-commons-io, install SRPM into chroot with rpm -Uvh and build the SRPM with rpmbuild -bb.

Comment 10 Alexander Todorov 2014-03-07 09:37:11 UTC
(In reply to Mikolaj Izdebski from comment #7)

> Isn't LiceCD using some unionfs/aufs to merge contents of tmpfs and iso9660?
> That could be a file system limitation.
>

I'm not rebuilding the package on LiveCD, I only used it for installing the system. 
 
> If you want this to be resolved you'll have to provide extra information:
> 
> 1) file system type and mount flags used (attach appriopriate line from
> /etc/mtab)
> 

# cat /etc/mtab | grep "/ "
rootfs / rootfs rw,seclabel 0 0
/dev/mapper/fedora_hp--ml370g6--01-root / ext4 rw,seclabel,relatime,data=ordered 0 0


> 2) full build logs with debugging output enabled (append -X to line
> containing %mvn_build)
> 
> 3) surefire test logs, available under target/surefire-reports/ after build
> fails

Comment 11 Alexander Todorov 2014-03-07 09:37:53 UTC
Created attachment 871810 [details]
build log with debug enabled

Comment 12 Alexander Todorov 2014-03-07 09:38:07 UTC
Created attachment 871811 [details]
surefire-reports.tar.gz

Comment 13 Mikolaj Izdebski 2014-03-07 09:40:14 UTC
Thank you, I'll look into it.

Comment 14 Mikolaj Izdebski 2014-03-07 10:06:28 UTC
You shouldn't be rebuilding the package as root.  Try rebuilding the package as unpriviledged user.

Comment 15 Michal Srb 2014-03-07 11:39:30 UTC
Ok, I know how to reproduce it. If you first try to build the package as a root, it will create some test files in /tmp/tmp-FileUtilsTestCase/ and these temporary files will stay there (created by root). If you then try to rebuild the package as non-root user, it will fail with "java.io.IOException: Unable to delete file: /tmp/tmp-FileUtilsTestCase/sub/A.txt" and similar.

Tests aren't perfect, but IMO this is not a bug. Everything works correctly, if you build the package as non-root user.

Comment 16 Alexander Todorov 2014-03-07 14:43:06 UTC
(In reply to Michal Srb from comment #15)
> Ok, I know how to reproduce it. If you first try to build the package as a
> root, it will create some test files in /tmp/tmp-FileUtilsTestCase/ and
> these temporary files will stay there (created by root). If you then try to
> rebuild the package as non-root user, it will fail with
> "java.io.IOException: Unable to delete file:
> /tmp/tmp-FileUtilsTestCase/sub/A.txt" and similar.
> 


I still see to issues here regardless if you run as root or non-root: 

1) Tests should be using the equivalent of mktemp -d and not rely on a fixed location and
2) Tests should clean up properly after themselves


Additional info:
After doing rm -rf /tmp/tmp-FileUtilsTestCase/ and rebuilding the package twice as non-root all works as expected.


After rm -rf /tmp/tmp-FileUtilsTestCase/ and rebuilding as ROOT I still get an error: 

Failed tests: 
  FileFilterTestCase.testCanRead:847->assertFiltering:65 Filter(File) org.apache.commons.io.filefilter.AndFileFilter not true for /root/rpmbuild/BUILD/commons-io-2.4-src/test/io/read-only-file1.txt expected:<true> but was:<false>
  FileFilterTestCase.testCanWrite:857->assertFiltering:65 Filter(File) org.apache.commons.io.filefilter.CanWriteFileFilter not false for /root/rpmbuild/BUILD/commons-io-2.4-src/test/io/read-only-file2.txt expected:<false> but was:<true>
  FileUtilsCleanDirectoryTestCase.testThrowsOnNullList:102 expected IOException
  FileUtilsCleanDirectoryTestCase.testThrowsOnCannotDeleteFile:121 expected IOException



Note that this error doesn't appear to have anything to do with /tmp/tmp-FileUtilsTestCase/ in the sense that it prints other file paths which don't exist. 

Also in this second case /tmp/tmp-FileUtilsTestCase/ is owned by root so permissions shouldn't be a problem. 


Any explanation to why is that?

Comment 17 Mikolaj Izdebski 2014-03-07 14:55:13 UTC
(In reply to Alexander Todorov from comment #16)
> 1) Tests should be using the equivalent of mktemp -d and not rely on a fixed
> location and

Agreed, but unfortunately that's not the case.

> 2) Tests should clean up properly after themselves

Agreed.

You are free to send patches upstream. I don't have time for that and current tests work for me as package owner. They clearly work for upstream too.

> Any explanation to why is that?

Basically there were 2 reasons for test failures:

1) When ran as *root* some tests were failing because they are assuming they are ran as non-root.  To be more specific, failing test cases were checking if attempts to read from or write to files without appropriate permissions throws an exception.  Because root has access to any files (no matter of inode access mode), no exception was thrown, which was causing the tests were failure.

2) When ran as *non-root* some tests were failing because they were unable to remove pre-existing directory in /tmp which owned by root.  This directory was created by previous test runs as root.

The first case can't be easily fixed, the solution is not to run tests as root.  The second case indicates rather poor design of tests and not bugs in the actual code.


Note You need to log in before you can comment on or make changes to this bug.