Bug 1088966 - crash during any build on el6 and el7
Summary: crash during any build on el6 and el7
Keywords:
Status: NEW
Alias: None
Product: Fedora EPEL
Classification: Fedora
Component: mock
Version: epel7
Hardware: Unspecified
OS: Unspecified
low
unspecified
Target Milestone: ---
Assignee: Miroslav Suchý
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-04-17 14:19 UTC by Levente Farkas
Modified: 2017-02-22 08:53 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-05-23 20:42:46 UTC


Attachments (Terms of Use)
site-defaults.cfg (11.70 KB, text/plain)
2015-04-24 11:47 UTC, Levente Farkas
no flags Details
viduxos-7-x86_64.cfg (2.29 KB, text/plain)
2015-04-24 11:48 UTC, Levente Farkas
no flags Details

Description Levente Farkas 2014-04-17 14:19:48 UTC
crash during any build with:
---------------------------------
Start: build setup for java-1.6.0-openjdk-1.6.0.0-5.1.13.3.el6_5.src.rpm
INFO: unmounting tmpfs.
ERROR: Exception(java-1.6.0-openjdk-1.6.0.0-5.1.13.3.el6_5.src.rpm) Config(os-6-x86_64) 0 minutes 3 seconds
INFO: Results and/or logs in: /var/lib/mock/os-6-x86_64/result
ERROR: [Errno 2] No such file or directory: 'java-1.6.0-openjdk-1.6.0.0-5.1.13.3.el6_5.src.rpm'
Traceback (most recent call last):
  File "/usr/sbin/mock", line 696, in <module>
    main(retParams)
  File "/usr/sbin/mock", line 630, in main
    do_rebuild(config_opts, chroot, args)
  File "<peak.util.decorators.rewrap wrapping __main__.do_rebuild at 0x025F3668>", line 3, in do_rebuild
    def do_rebuild(config_opts, chroot, srpms): return __decorated(config_opts, chroot, srpms)
  File "/usr/lib/python2.6/site-packages/mockbuild/trace_decorator.py", line 70, in trace
    result = func(*args, **kw)
  File "/usr/sbin/mock", line 278, in do_rebuild
    chroot.build(srpm, timeout=config_opts['rpmbuild_timeout'], check=config_opts['check'])
  File "<peak.util.decorators.rewrap wrapping mockbuild.backend.build at 0x025EE140>", line 3, in build
    def build(self, srpm, timeout, check): return __decorated(self, srpm, timeout, check)
  File "/usr/lib/python2.6/site-packages/mockbuild/trace_decorator.py", line 70, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.6/site-packages/mockbuild/backend.py", line 689, in build
    srpmChrootFilename = self._copySrpmIntoChroot(srpm)
  File "<peak.util.decorators.rewrap wrapping mockbuild.backend._copySrpmIntoChroot at 0x025F5AA0>", line 3, in _copySrpmIntoChroot
    def _copySrpmIntoChroot(self, srpm): return __decorated(self, srpm)
  File "/usr/lib/python2.6/site-packages/mockbuild/trace_decorator.py", line 70, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.6/site-packages/mockbuild/backend.py", line 1092, in _copySrpmIntoChroot
    shutil.copy2(srpm, dest)
  File "/usr/lib64/python2.6/shutil.py", line 95, in copy2
    copyfile(src, dst)
  File "/usr/lib64/python2.6/shutil.py", line 50, in copyfile
    with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: 'java-1.6.0-openjdk-1.6.0.0-5.1.13.3.el6_5.src.rpm'
---------------------------------

Comment 1 Levente Farkas 2014-05-16 08:47:40 UTC
any progress with it?

Comment 2 Jon Disnard 2014-05-22 17:19:34 UTC
Can you see if the copyin or copyout features of mock are working independently, instead of performing a build from srpm? The part of the bottom with IO error is what I'm seeking to findout. Thanks

Comment 3 Levente Farkas 2014-05-22 17:35:27 UTC
we've got already a long conversation with clark about this but and after he can't find the reason told me to write this bugzilla.

what should i've to test exactly?

Comment 4 Clark Williams 2014-05-22 18:45:48 UTC
 What we need is a way to force this failure so that we can debug it. I just built some (simple) srpms using 1.1.38 on both f20 and rhel6.5, using the epel-6-x86_64 config and didn't get any failures. 

I think we need to see the command line you used to start mock and the build and root logs from the failed build.  Oh and have you modified any of the configuration files?

Comment 5 Levente Farkas 2014-05-23 11:28:15 UTC
yesterday we give up and reinstall all of our build servers to rhel7 in a hope that is solves our problems. so unfortunately i'm no longer be able to test it again. but from our earlier conversation:

------------------------------
same happened (actually i already try it before even with full path:-()

ERROR: [Errno 2] No such file or directory:
'./star-1.5-11.1.el6_5.src.rpm'
Traceback (most recent call last):
  File "/usr/sbin/mock", line 697, in <module>
    main(retParams)
  File "/usr/sbin/mock", line 631, in main
    do_rebuild(config_opts, chroot, args)
  File "<peak.util.decorators.rewrap wrapping __main__.do_rebuild at
0x027EE668>", line 3, in do_rebuild
    def do_rebuild(config_opts, chroot, srpms): return
__decorated(config_opts, chroot, srpms)
  File
"/usr/lib/python2.6/site-packages/mockbuild/trace_decorator.py", line
70, in trace
    result = func(*args, **kw)
  File "/usr/sbin/mock", line 279, in do_rebuild
    chroot.build(srpm, timeout=config_opts['rpmbuild_timeout'],
check=config_opts['check'])
  File "<peak.util.decorators.rewrap wrapping mockbuild.backend.build
at 0x027E90C8>", line 3, in build
    def build(self, srpm, timeout, check): return __decorated(self,
srpm, timeout, check)
  File
"/usr/lib/python2.6/site-packages/mockbuild/trace_decorator.py", line
70, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.6/site-packages/mockbuild/backend.py", line
688, in build
    srpmChrootFilename = self._copySrpmIntoChroot(srpm)
  File "<peak.util.decorators.rewrap wrapping
mockbuild.backend._copySrpmIntoChroot at 0x027F0A28>", line 3, in
_copySrpmIntoChroot
    def _copySrpmIntoChroot(self, srpm): return __decorated(self, srpm)
  File
"/usr/lib/python2.6/site-packages/mockbuild/trace_decorator.py", line
70, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.6/site-packages/mockbuild/backend.py", line
1091, in _copySrpmIntoChroot
    shutil.copy2(srpm, dest)
  File "/usr/lib64/python2.6/shutil.py", line 95, in copy2
    copyfile(src, dst)
  File "/usr/lib64/python2.6/shutil.py", line 50, in copyfile
    with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory:
'./star-1.5-11.1.el6_5.src.rpm'
------------------------------

Comment 6 Jon Disnard 2014-05-23 20:42:46 UTC
(In reply to Levente Farkas from comment #5)
> yesterday we give up and reinstall all of our build servers to rhel7 in a
> hope that is solves our problems. so unfortunately i'm no longer be able to
> test it again. but from our earlier conversation:

...


Alright well good luck, and plz feel free to reopen the ticket if you hit this again. Would be happy to help.

Comment 7 Levente Farkas 2014-05-26 08:28:53 UTC
now it seems also happened on rhel7. so i reopen it. the details:
- rhel7 rc1
- kernel-3.10.0-121.el7.x86_64  
- mock-1.1.38-1.el7.noarch

# ls -al a2ps-4.14-23.el7.src.rpm 
lrwxrwxrwx 1 root root 96 May 22 11:24 a2ps-4.14-23.el7.src.rpm -> /mnt/download/mirror/redhat-7/Workstation-optional/source/tree/Packages/a2ps-4.14-23.el7.src.rpm

state.log:
2014-05-26 10:19:01,365 - Mock Version: 1.1.38
2014-05-26 10:19:01,370 - Start: unpacking root cache
2014-05-26 10:19:03,097 - Finish: unpacking root cache
2014-05-26 10:19:03,098 - Start: cleaning yum metadata
2014-05-26 10:19:03,103 - Finish: cleaning yum metadata
2014-05-26 10:19:03,110 - Start: device setup
2014-05-26 10:19:03,112 - Finish: device setup
2014-05-26 10:19:03,112 - Start: yum update
2014-05-26 10:19:12,784 - Finish: yum update
2014-05-26 10:19:12,837 - Finish: lock buildroot
2014-05-26 10:19:12,837 - Finish: chroot init
2014-05-26 10:19:12,918 - Start: build phase for a2ps-4.14-23.el7.src.rpm
2014-05-26 10:19:12,918 - Start: device setup
2014-05-26 10:19:12,921 - Finish: device setup
2014-05-26 10:19:12,939 - Start: build setup for a2ps-4.14-23.el7.src.rpm

so build log is still empty:-(

Starting at Mon May 26 10:19:01 CEST 2014: mock -r viduxos-7-x86_64 --define "rhel 7" --define ".el7 1" --define "dist .el7" --rebuild a2ps-4.14-23.el7.src.rpm
INFO: mock.py version 1.1.38 starting...
Start: init plugins
INFO: tmpfs initialized
INFO: selinux disabled
Finish: init plugins
Start: run
INFO: Start(a2ps-4.14-23.el7.src.rpm)  Config(viduxos-7-x86_64)
Start: lock buildroot
Start: clean chroot
INFO: chroot (/var/lib/mock/viduxos-7-x86_64) unlocked and deleted
Finish: clean chroot
Finish: lock buildroot
Start: chroot init
Start: lock buildroot
Mock Version: 1.1.38
INFO: Mock Version: 1.1.38
INFO: calling preinit hooks
INFO: mounting tmpfs at /var/lib/mock/viduxos-7-x86_64/root/.
INFO: enabled root cache
Start: unpacking root cache
Finish: unpacking root cache
INFO: enabled yum cache
Start: cleaning yum metadata
Finish: cleaning yum metadata
INFO: enabled ccache
Start: device setup
Finish: device setup
Start: yum update
Finish: yum update
Finish: lock buildroot
Finish: chroot init
INFO: Installed packages:
Start: build phase for a2ps-4.14-23.el7.src.rpm
Start: device setup
Finish: device setup
Start: build setup for a2ps-4.14-23.el7.src.rpm
INFO: unmounting tmpfs.
ERROR: Exception(a2ps-4.14-23.el7.src.rpm) Config(viduxos-7-x86_64) 0 minutes 11 seconds
INFO: Results and/or logs in: /var/lib/mock/viduxos-7-x86_64/result
ERROR: [Errno 2] No such file or directory: 'a2ps-4.14-23.el7.src.rpm'
Traceback (most recent call last):
  File "/usr/sbin/mock", line 306, in <module>
    def do_buildsrpm(config_opts, chroot, options, args):
  File "/usr/sbin/mock", line 630, in main
    do_rebuild(config_opts, chroot, args)
  File "<peak.util.decorators.rewrap wrapping __main__.do_rebuild at 0x00EB8B18>", line 3, in do_rebuild
    def do_rebuild(config_opts, chroot, srpms): return __decorated(config_opts, chroot, srpms)
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 70, in trace
    result = func(*args, **kw)
  File "/usr/sbin/mock", line 278, in do_rebuild
    chroot.build(srpm, timeout=config_opts['rpmbuild_timeout'], check=config_opts['check'])
  File "<peak.util.decorators.rewrap wrapping mockbuild.backend.build at 0x00EB8C80>", line 3, in build
    def build(self, srpm, timeout, check): return __decorated(self, srpm, timeout, check)
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 70, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.7/site-packages/mockbuild/backend.py", line 689, in build
    srpmChrootFilename = self._copySrpmIntoChroot(srpm)
  File "<peak.util.decorators.rewrap wrapping mockbuild.backend._copySrpmIntoChroot at 0x00EB9E60>", line 3, in _copySrpmIntoChroot
    def _copySrpmIntoChroot(self, srpm): return __decorated(self, srpm)
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 70, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.7/site-packages/mockbuild/backend.py", line 1092, in _copySrpmIntoChroot
    shutil.copy2(srpm, dest)
  File "/usr/lib64/python2.7/shutil.py", line 130, in copy2
    copyfile(src, dst)
  File "/usr/lib64/python2.7/shutil.py", line 82, in copyfile
    with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: 'a2ps-4.14-23.el7.src.rpm'

Comment 8 Levente Farkas 2014-05-26 10:23:12 UTC
i even try to copy the src.rpm to the current working dir and just run:

mock -r viduxos-7-x86_64 --define "rhel 7" --define ".el7 1" --define "dist .el7" --rebuild a2ps-4.14-23.el7.src.rpm

and the same happened.

so finally find the root of the problem. when change something in site-defaults.cfg then mock should have to rebuild the cache. if i set it then the first run working ie when the system rebuild the cache. but the second run when start to use it than it gives the above error.

Comment 9 Levente Farkas 2014-05-26 19:33:16 UTC
is there any other help how i can help...log debug version etc...?

Comment 10 Levente Farkas 2014-05-29 12:25:15 UTC
it's also happened on with the latest 1.1.39 version on rhel7. i don't know how can it be possible that it's working for others!?

Comment 11 Levente Farkas 2014-05-30 13:54:51 UTC
as i set 
config_opts['plugin_conf']['root_cache_enable'] = False
now everything works, but of course much more slower the before. so this's the worst workaround...

Comment 12 Levente Farkas 2014-06-03 13:28:13 UTC
can i get some kind of debug version to find this bug? it's really slow down or build times. i'd be happy to test any kind of solution or tips!

Comment 13 Levente Farkas 2014-07-14 22:46:59 UTC
is there any plan to be able to use root_cache_enable?

Comment 14 Levente Farkas 2014-07-20 19:46:48 UTC
in 1.1.40 root_cache support still broken. this slow down build process so much! any chance to look into this problem?

Comment 15 Miroslav Suchý 2014-11-18 18:33:47 UTC
Taking.

When site-default.cfg is updated then cache is invalidated. You should see:
INFO: enabled root cache
INFO: /etc/mock/site-defaults.cfg newer than root cache; cache will be rebuilt
INFO: enabled yum cache

That part is there since 2012.

Is it possible that your run mock from / So you are hitting 
https://bugzilla.redhat.com/show_bug.cgi?id=1160428#c12

Can you please try with mock-1.2.2 (in updates-testing right now).

Comment 16 Levente Farkas 2014-11-19 11:12:51 UTC
the problem still exists!
so the root_cache still not working. which would be a very good thing to use!!! since currently all builds are very slow!

Start: build phase for virt-who-0.8-15.el7_0.src.rpm
Start: build setup for virt-who-0.8-15.el7_0.src.rpm
ERROR: Exception(virt-who-0.8-15.el7_0.src.rpm) Config(viduxos-7-x86_64) 0 minutes 11 seconds
INFO: Results and/or logs in: /var/lib/mock/viduxos-7-x86_64/result
INFO: unmounting tmpfs.
ERROR: [Errno 2] No such file or directory: 'virt-who-0.8-15.el7_0.src.rpm'
Traceback (most recent call last):
  File "/usr/sbin/mock", line 797, in <module>
    main()
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/usr/sbin/mock", line 628, in main
    run_command(options, args, config_opts, commands, buildroot, state)
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/usr/sbin/mock", line 703, in run_command
    do_rebuild(config_opts, commands, buildroot, args)
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/usr/sbin/mock", line 474, in do_rebuild
    post=createrepo_on_rpms, clean=clean)
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/usr/sbin/mock", line 427, in rebuild_generic
    ret = cmd(item)
  File "/usr/sbin/mock", line 462, in build
    check=config_opts['check'])
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.7/site-packages/mockbuild/backend.py", line 210, in build
    srpm = self.copy_srpm_into_chroot(srpm)
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.7/site-packages/mockbuild/backend.py", line 363, in copy_srpm_into_chroot
    shutil.copy2(srpm_path, dest)
  File "/usr/lib64/python2.7/shutil.py", line 130, in copy2
    copyfile(src, dst)
  File "/usr/lib64/python2.7/shutil.py", line 82, in copyfile
    with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: 'virt-who-0.8-15.el7_0.src.rpm'

Comment 17 Miroslav Suchý 2014-11-24 13:20:18 UTC
I am using root_cache every day without problem.
Therefore I would say there is some problem related to your config.
Either please attach your config (both viduxos-7-x86_64.cfg and site-defaults.cfg) or reproduce that with default mock configs.

Comment 18 Miroslav Suchý 2015-04-24 06:39:52 UTC
Please provide:
 * /etc/mock/site-defaults.cfg
 * your config viduxos-7-x86_64
so I can reproduce it.

Comment 19 Levente Farkas 2015-04-24 11:47:54 UTC
Created attachment 1018430 [details]
site-defaults.cfg

Comment 20 Levente Farkas 2015-04-24 11:48:13 UTC
Created attachment 1018431 [details]
viduxos-7-x86_64.cfg

Comment 21 Levente Farkas 2015-04-24 11:49:12 UTC
and the reason of the bug is the 
config_opts['plugin_conf']['root_cache_enable'] = True
line in the config file. if i change it to False then it's working (although much more slower).

Comment 22 Miroslav Suchý 2017-02-21 16:21:09 UTC
Looking at this BZ again. 

Well, it seem that the src.rpm exist at the start (otherwise there would be another error earlier), so it must be deleted once you start building.

I assume you did not delete it yourself. So it must be in some directory which mock wipe out.

Is there any chance that you run:

mock --rebuild /var/lib/mock/viduxos-7-x86_64/results/foo.src.rpm

?

Because that would explain that thing and will be duplicate of bug 1392137

Comment 23 Levente Farkas 2017-02-22 08:53:05 UTC
not it's in an external directory ie. in our robot builder's home


Note You need to log in before you can comment on or make changes to this bug.