Bug 483486 - Can't build 'nosrc' srpms using mock (--no-clean does not work)
Can't build 'nosrc' srpms using mock (--no-clean does not work)
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: mock (Show other bugs)
19
x86_64 Linux
medium Severity medium
: ---
: ---
Assigned To: Miroslav Suchý
Fedora Extras Quality Assurance
:
Depends On: 1166121
Blocks:
  Show dependency treegraph
 
Reported: 2009-02-01 13:13 EST by Alex
Modified: 2016-02-23 08:24 EST (History)
17 users (show)

See Also:
Fixed In Version: mock-1.2.3-1.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-12-11 23:07:30 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Keep the build directory when --no-clean specified (1.35 KB, patch)
2013-10-24 17:58 EDT, Clark Williams
no flags Details | Diff

  None (edit)
Description Alex 2009-02-01 13:13:26 EST
Description of problem: Can't build 'nosrc' srpms from JPackage for a 32-bit target using mock, on a 64-bit host.


Version-Release number of selected component (if applicable): 0.9.13-1.fc10


How reproducible: Every time
 

Steps to Reproduce:

1. Download a 'nosrc' srpm from JPackage, e.g. 'java-1.6.0-sun':
 
'wget http://mirrors.dotsrc.org/jpackage/5.0/generic/non-free/SRPMS/java-1.6.0-sun-1.6.0.11-1jpp.nosrc.rpm'

2. Download the 32-bit 'bin' file from Sun, corresponding to the version of the 'nosrc' rpm:

'wget -O jdk-6u11-linux-i586.bin "http://cds.sun.com/is-bin/INTERSHOP.enfinity/WFS/CDS-CDS_Developer-Site/en_US/-/USD/VerifyItem-Start/jdk-6u11-linux-i586.bin?BundledLineItemUUID=qz5IBe.phWsAAAEf6CEk4zLq&OrderID=kzJIBe.pYgYAAAEf2iEk4zLq&ProductID=abtIBe.ovJUAAAEdeDBGb7Et&FileName=/jdk-6u11-linux-i586.bin"'

3. Init the mock chroot:

'mock -v -r fedora-10-i386 --init'

4. Use mock to copy in the bin file to the SOURCES directory in the chroot: 

'mock -v -r fedora-10-i386 --copyin jdk-6u11-linux-i586.bin /builddir/build/SOURCES/'

5. Confirm the bin file was copied into the chroot:

ls -l '/var/lib/mock/fedora-10-i386/root/builddir/build/SOURCES/'

6. Use mock to rebuild the srpm, with the '--no-clean' option:

'mock -v -r fedora-10-i386 --no-clean --rebuild java-1.6.0-sun-1.6.0.11-1jpp.nosrc.rpm'
  
Actual results:

Mock deletes the SOURCES directory so the bin file can't be found, even though the '--no-clean' option was specified:

DEBUG: remove tree: /var/lib/mock/fedora-10-i386/root/builddir

Expected results:

Mock doesn't delete the SOURCES directory and the RPM files are built successfully.

Additional info:
Comment 1 Chandler Wilkerson 2011-10-04 15:25:04 EDT
I can confirm this is still not fixed for mock 1.1.12 in epel.

Despite using the --no-clean flag, the --rebuild command removes the mockbuild user and its home directory /builddir then recreates the user. The verbose flag doesn't show where the srpm gets copied back into the /builddir/build/originals directory.

I'd like to use nosrc rpms with mock to handle my university's build of various proprietary packages that we can only legally handle as nosrc rpms (maple, mathematica, matlab)

--
Chandler Wilkerson
Rice University
Comment 2 Jessica Jones 2012-11-30 06:35:13 EST
I can confirm that this bug is still present on an up-to-date Fedora 16.  I am upgrading to Fedora 17 and should be able to confirm if it is present there.

As I am trying to package several nosrc RPMs, I would really appreciate this being fixed soon.  It is not possible to build nosrc RPMs using mock at present.
Comment 3 Jessica Jones 2012-11-30 12:38:32 EST
(In reply to comment #2)
> I can confirm that this bug is still present on an up-to-date Fedora 16.  I
> am upgrading to Fedora 17 and should be able to confirm if it is present
> there.
> 

This is still a problem in Fedora 17 (1.1.28).
Comment 4 Clark Williams 2012-12-03 14:28:55 EST
Jessica, 

do you have a couple of the nosrc SRPMS handy? If so would you please attach them to this BZ so I can use them for testing?
Comment 5 Chandler Wilkerson 2012-12-03 15:05:05 EST
Clark,

For testing, you could just pick up a regular SRPM, add nosource: 1 to the .spec file (where 1 is the primary .tar.bz source file) and test with that.

In my experience, nosrc rpms tend to be too large and unwieldy for testing.
Comment 6 Jessica Jones 2013-03-07 09:34:13 EST
(In reply to comment #4)
> Jessica, 
> 
> do you have a couple of the nosrc SRPMS handy? If so would you please attach
> them to this BZ so I can use them for testing?

http://mirrors.dotsrc.org/jpackage/5.0/generic/non-free/SRPMS/

has several.
Comment 7 Clark Williams 2013-10-24 17:58:15 EDT
Created attachment 815938 [details]
Keep the build directory when --no-clean specified

This is a first step in trying to sort out building nosrc srpms
Comment 8 Miro Hrončok 2013-12-20 17:01:27 EST
bump?
Comment 9 Miroslav Suchý 2014-11-18 11:10:18 EST
Fedorahosted product is retired and nearly invisible. Moving to Fedora product.
Comment 10 Michael Simacek 2014-11-19 07:20:10 EST
Not deleting builddir could be easily implemented in the current version of mock and would actually simplify the code (the part of code has already been isolated for the sake of having --short-circuit). But I think users expect the build directory to be deleted. At least I always thought of --no-clean as an option that let's me build the package without the need to reinstall it's buildrequires, but still having reproducible build (minus the BR part ofc). If we don't delete the build dir there will be garbage left behind the previous build that will affect the build.
And mock will need to get a better way to know which files were output by rpmbuild than globbing (and I'm not sure if there is a better way). Otherwise you wouldn't be able to build two different SRPMs in a row:
ERROR: Expected to find single rebuilt srpm, found 2.

So if this is still necessary, I'd suggest having some other option for this than --no-clean, such as --no-delete-builddir.
Comment 11 Miroslav Suchý 2014-11-19 08:57:32 EST
Well - from man page:
       --rebuild
              If  no  command  is specified, rebuild is assumed. Rebuilds the specified SRPM(s). The buildroot is cleaned
              first, unless --no-clean is specified.
       -n, --no-clean
              Do not clean chroot before building package.
Comment 12 Miroslav Suchý 2014-11-19 10:27:14 EST
The SOURCE dir is actually deleted when:
> /home/msuchy/projects/mock/py/mockbuild/buildroot.py(216)_make_build_user()
-> self.doChroot(['/usr/sbin/userdel', '-r', '-f', dets['user']], shell=False, raiseExc=False)
because that delete home dir, which is /builddir

I would use:
-        self.doChroot(['/usr/sbin/userdel', '-r', '-f', dets['user']], shell=False, raiseExc=False)
+        if self.config['clean']:
+            self.doChroot(['/usr/sbin/userdel', '-r', '-f', dets['user']], shell=False, raiseExc=False)
+        else:
+            self.doChroot(['/usr/sbin/userdel', '-f', dets['user']], shell=False, raiseExc=False)


But additionally, this:
> /home/msuchy/projects/mock/py/mockbuild/buildroot.py(360)_setup_build_dirs()
-> os.chown(os.path.join(dirpath, path), self.chrootuid, -1)
would fail as well, because --copyin create the files as root:root so mockbuild user can not chown them.

I am not sure if better would be copyin let create file as mockbuilder user or skip _setup_build_dirs() if --no-clean was used.
Comment 13 Miroslav Suchý 2014-11-19 11:19:07 EST
Hmm but then we have problems, because right after:
  mock -r fedora-20-i386 --init
is:
  $ ls -l '/var/lib/mock/fedora-20-i386/root/builddir/build/'
total 28
drwxr-xr-x 2 msuchy msuchy 4096 Nov 19 17:16 BUILD
drwxr-xr-x 2 msuchy msuchy 4096 Nov 19 17:16 BUILDROOT
drwxr-xr-x 2 msuchy msuchy 4096 Nov 19 17:16 originals
drwxr-xr-x 2 msuchy msuchy 4096 Nov 19 17:16 RPMS
drwxr-xr-x 2 msuchy msuchy 4096 Nov 19 17:16 SOURCES
drwxr-xr-x 2 msuchy msuchy 4096 Nov 19 17:16 SPECS
drwxr-xr-x 2 msuchy msuchy 4096 Nov 19 17:16 SRPMS

I guess that we should place that chown somewhere else too.
Comment 14 Miroslav Suchý 2014-11-19 11:19:34 EST
taking as I'm working on this.
Comment 15 Miroslav Suchý 2014-11-20 07:26:16 EST
Fixed in commit 49de45d.

But it will not work anyway until bug 1166121 is resolved.
Comment 16 Fedora Update System 2014-12-04 07:06:36 EST
mock-1.2.3-1.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/mock-1.2.3-1.fc21
Comment 17 Fedora Update System 2014-12-04 07:10:07 EST
mock-1.2.3-1.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/mock-1.2.3-1.fc20
Comment 18 Fedora Update System 2014-12-04 07:11:03 EST
mock-1.2.3-1.el7 has been submitted as an update for Fedora EPEL 7.
https://admin.fedoraproject.org/updates/mock-1.2.3-1.el7
Comment 19 Fedora Update System 2014-12-04 07:12:31 EST
mock-1.2.3-1.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/mock-1.2.3-1.el6
Comment 20 Fedora Update System 2014-12-04 19:46:47 EST
Package mock-1.2.3-1.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing mock-1.2.3-1.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-16285/mock-1.2.3-1.fc21
then log in and leave karma (feedback).
Comment 21 Petr Pisar 2014-12-09 08:56:54 EST
(In reply to Miroslav Suchý from comment #11)
> Well - from man page:
>        --rebuild
>               If  no  command  is specified, rebuild is assumed. Rebuilds
> the specified SRPM(s). The buildroot is cleaned
>               first, unless --no-clean is specified.
>        -n, --no-clean
>               Do not clean chroot before building package.

What's the difference between "buildroot" and "chroot"?
Comment 22 Miroslav Suchý 2014-12-10 04:17:31 EST
Good point. 

s/buildroot/chroot/ in commit 95849c2
Comment 23 Fedora Update System 2014-12-11 23:07:30 EST
mock-1.2.3-1.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 24 Fedora Update System 2014-12-16 23:42:33 EST
mock-1.2.3-1.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 25 Fedora Update System 2015-01-05 21:06:17 EST
mock-1.2.3-1.el7 has been pushed to the Fedora EPEL 7 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 26 Fedora Update System 2015-01-05 21:07:42 EST
mock-1.2.3-1.el6 has been pushed to the Fedora EPEL 6 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 27 praetorian 2016-02-18 13:43:51 EST
Has this actually been fixed? I ran into this bug in 1.2.14 on CentOS 6.7.

This is what I see in buildroot.py (in function _make_build_user) checked out from mock tip

        if self.config['clean']:
            self.doChroot(['/usr/sbin/userdel', '-r', '-f', dets['user']],
                          shell=False, raiseExc=False, nosync=True)
        else:
            self.doChroot(['/usr/sbin/userdel', '-f', dets['user']],
                          shell=False, raiseExc=False, nosync=True)
        self.doChroot(['/usr/sbin/userdel', '-r', '-f', dets['user']],
                      shell=False, raiseExc=False, nosync=True)


Note that the third "userdel -r" is executed unconditionally.
Comment 28 praetorian 2016-02-18 18:27:34 EST
Just to make sure I'm barking up the right bugzilla, what I want to have happen is for "/builddir/build/BUILD" under the chroot to persist between two calls to "mock --buildsrpm --no-clean" (I've run "mock --init" prior to the two buildsrpm calls).

The lines of code I pointed to in the previous are not what's deleting the builddir (although the last "userdel -r" call seems to be bug too), but it's the line immediately before that.

Line 248 of buildroot.py is

        util.rmtree(self.make_chroot_path(self.homedir),
                    selinux=self.selinux, exclude=excluded)

That unconditionally deletes /builddir under the chroot. It should probably be

        if self.config['clean']:
            util.rmtree(self.make_chroot_path(self.homedir),
                        selinux=self.selinux, exclude=excluded)

I can create a new bug report if that's the appropriate thing to do.
Comment 29 Michael Simacek 2016-02-19 03:23:55 EST
(In reply to praetorian from comment #28)
> Just to make sure I'm barking up the right bugzilla, what I want to have
> happen is for "/builddir/build/BUILD" under the chroot to persist between
> two calls to "mock --buildsrpm --no-clean" (I've run "mock --init" prior to
> the two buildsrpm calls).
> 
> The lines of code I pointed to in the previous are not what's deleting the
> builddir (although the last "userdel -r" call seems to be bug too), but it's
> the line immediately before that.
> 
> Line 248 of buildroot.py is
> 
>         util.rmtree(self.make_chroot_path(self.homedir),
>                     selinux=self.selinux, exclude=excluded)

Please note the "excluded" argument - it doesn't remove the whole tree, but keeps selected files/dirs present. It's configured in config_opts['exclude_from_homedir_cleanup'] - default to ['build/SOURCES'], so this should make nosrc rpm builds possible.
Why do you want to keep BUILD directory?
Comment 30 praetorian 2016-02-19 20:23:42 EST
(In reply to Michael Simacek from comment #29)
> Please note the "excluded" argument - it doesn't remove the whole tree, but
> keeps selected files/dirs present. It's configured in
> config_opts['exclude_from_homedir_cleanup'] - default to ['build/SOURCES'],
> so this should make nosrc rpm builds possible.

Thank you Michael! Setting "config_opts['exclude_from_homedir_cleanup'] = ['build/SOURCES', 'build/BUILD']" in my mock config file does what I want. While looking for information on that, I also realized that site-defaults.cfg has documentation for all parameters, that should come in handy.

> Why do you want to keep BUILD directory?

I'm trying to create a single software collection (scl) from the latest fc24 SRPMs for LLVM and clang. Turns out if I want to run "make check" on the clang build, I need the original LLVM build tree because that contains a unit test framework that clang uses too. Preserving the BUILD directory seems like the easiest way to do this.

Sorry to keep bugging you about this, but isn't the third call to "userdel" in comment #27 a bug?

In the first two calls, you conditionally call "userdel -r" or "userdel" depending on "config['clean']", but then you call "userdel -r" anyway right afterwards.
Comment 31 Michael Simacek 2016-02-23 08:24:25 EST
(In reply to praetorian from comment #30)
> (In reply to Michael Simacek from comment #29)
> > Please note the "excluded" argument - it doesn't remove the whole tree, but
> > keeps selected files/dirs present. It's configured in
> > config_opts['exclude_from_homedir_cleanup'] - default to ['build/SOURCES'],
> > so this should make nosrc rpm builds possible.
> 
> Thank you Michael! Setting "config_opts['exclude_from_homedir_cleanup'] =
> ['build/SOURCES', 'build/BUILD']" in my mock config file does what I want.
> While looking for information on that, I also realized that
> site-defaults.cfg has documentation for all parameters, that should come in
> handy.
> 
> > Why do you want to keep BUILD directory?
> 
> I'm trying to create a single software collection (scl) from the latest fc24
> SRPMs for LLVM and clang. Turns out if I want to run "make check" on the
> clang build, I need the original LLVM build tree because that contains a
> unit test framework that clang uses too. Preserving the BUILD directory
> seems like the easiest way to do this.

The correct way to do that would be to package that test framework in a subpackage and buildrequire it in clang.

> 
> Sorry to keep bugging you about this, but isn't the third call to "userdel"
> in comment #27 a bug?

Kind of. It always fails because the user has been deleted already, so it does nothing. I'll prepare a patch that removes it.

Note You need to log in before you can comment on or make changes to this bug.