any progress with it?

Can you see if the copyin or copyout features of mock are working independently, instead of performing a build from srpm? The part of the bottom with IO error is what I'm seeking to findout. Thanks

we've got already a long conversation with clark about this but and after he can't find the reason told me to write this bugzilla.

what should i've to test exactly?

 What we need is a way to force this failure so that we can debug it. I just built some (simple) srpms using 1.1.38 on both f20 and rhel6.5, using the epel-6-x86_64 config and didn't get any failures. 

I think we need to see the command line you used to start mock and the build and root logs from the failed build.  Oh and have you modified any of the configuration files?

yesterday we give up and reinstall all of our build servers to rhel7 in a hope that is solves our problems. so unfortunately i'm no longer be able to test it again. but from our earlier conversation:

------------------------------
same happened (actually i already try it before even with full path:-()

ERROR: [Errno 2] No such file or directory:
'./star-1.5-11.1.el6_5.src.rpm'
Traceback (most recent call last):
  File "/usr/sbin/mock", line 697, in <module>
    main(retParams)
  File "/usr/sbin/mock", line 631, in main
    do_rebuild(config_opts, chroot, args)
  File "<peak.util.decorators.rewrap wrapping __main__.do_rebuild at
0x027EE668>", line 3, in do_rebuild
    def do_rebuild(config_opts, chroot, srpms): return
__decorated(config_opts, chroot, srpms)
  File
"/usr/lib/python2.6/site-packages/mockbuild/trace_decorator.py", line
70, in trace
    result = func(*args, **kw)
  File "/usr/sbin/mock", line 279, in do_rebuild
    chroot.build(srpm, timeout=config_opts['rpmbuild_timeout'],
check=config_opts['check'])
  File "<peak.util.decorators.rewrap wrapping mockbuild.backend.build
at 0x027E90C8>", line 3, in build
    def build(self, srpm, timeout, check): return __decorated(self,
srpm, timeout, check)
  File
"/usr/lib/python2.6/site-packages/mockbuild/trace_decorator.py", line
70, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.6/site-packages/mockbuild/backend.py", line
688, in build
    srpmChrootFilename = self._copySrpmIntoChroot(srpm)
  File "<peak.util.decorators.rewrap wrapping
mockbuild.backend._copySrpmIntoChroot at 0x027F0A28>", line 3, in
_copySrpmIntoChroot
    def _copySrpmIntoChroot(self, srpm): return __decorated(self, srpm)
  File
"/usr/lib/python2.6/site-packages/mockbuild/trace_decorator.py", line
70, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.6/site-packages/mockbuild/backend.py", line
1091, in _copySrpmIntoChroot
    shutil.copy2(srpm, dest)
  File "/usr/lib64/python2.6/shutil.py", line 95, in copy2
    copyfile(src, dst)
  File "/usr/lib64/python2.6/shutil.py", line 50, in copyfile
    with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory:
'./star-1.5-11.1.el6_5.src.rpm'
------------------------------

(In reply to Levente Farkas from comment #5)
> yesterday we give up and reinstall all of our build servers to rhel7 in a
> hope that is solves our problems. so unfortunately i'm no longer be able to
> test it again. but from our earlier conversation:

...


Alright well good luck, and plz feel free to reopen the ticket if you hit this again. Would be happy to help.

now it seems also happened on rhel7. so i reopen it. the details:
- rhel7 rc1
- kernel-3.10.0-121.el7.x86_64  
- mock-1.1.38-1.el7.noarch

# ls -al a2ps-4.14-23.el7.src.rpm 
lrwxrwxrwx 1 root root 96 May 22 11:24 a2ps-4.14-23.el7.src.rpm -> /mnt/download/mirror/redhat-7/Workstation-optional/source/tree/Packages/a2ps-4.14-23.el7.src.rpm

state.log:
2014-05-26 10:19:01,365 - Mock Version: 1.1.38
2014-05-26 10:19:01,370 - Start: unpacking root cache
2014-05-26 10:19:03,097 - Finish: unpacking root cache
2014-05-26 10:19:03,098 - Start: cleaning yum metadata
2014-05-26 10:19:03,103 - Finish: cleaning yum metadata
2014-05-26 10:19:03,110 - Start: device setup
2014-05-26 10:19:03,112 - Finish: device setup
2014-05-26 10:19:03,112 - Start: yum update
2014-05-26 10:19:12,784 - Finish: yum update
2014-05-26 10:19:12,837 - Finish: lock buildroot
2014-05-26 10:19:12,837 - Finish: chroot init
2014-05-26 10:19:12,918 - Start: build phase for a2ps-4.14-23.el7.src.rpm
2014-05-26 10:19:12,918 - Start: device setup
2014-05-26 10:19:12,921 - Finish: device setup
2014-05-26 10:19:12,939 - Start: build setup for a2ps-4.14-23.el7.src.rpm

so build log is still empty:-(

Starting at Mon May 26 10:19:01 CEST 2014: mock -r viduxos-7-x86_64 --define "rhel 7" --define ".el7 1" --define "dist .el7" --rebuild a2ps-4.14-23.el7.src.rpm
INFO: mock.py version 1.1.38 starting...
Start: init plugins
INFO: tmpfs initialized
INFO: selinux disabled
Finish: init plugins
Start: run
INFO: Start(a2ps-4.14-23.el7.src.rpm)  Config(viduxos-7-x86_64)
Start: lock buildroot
Start: clean chroot
INFO: chroot (/var/lib/mock/viduxos-7-x86_64) unlocked and deleted
Finish: clean chroot
Finish: lock buildroot
Start: chroot init
Start: lock buildroot
Mock Version: 1.1.38
INFO: Mock Version: 1.1.38
INFO: calling preinit hooks
INFO: mounting tmpfs at /var/lib/mock/viduxos-7-x86_64/root/.
INFO: enabled root cache
Start: unpacking root cache
Finish: unpacking root cache
INFO: enabled yum cache
Start: cleaning yum metadata
Finish: cleaning yum metadata
INFO: enabled ccache
Start: device setup
Finish: device setup
Start: yum update
Finish: yum update
Finish: lock buildroot
Finish: chroot init
INFO: Installed packages:
Start: build phase for a2ps-4.14-23.el7.src.rpm
Start: device setup
Finish: device setup
Start: build setup for a2ps-4.14-23.el7.src.rpm
INFO: unmounting tmpfs.
ERROR: Exception(a2ps-4.14-23.el7.src.rpm) Config(viduxos-7-x86_64) 0 minutes 11 seconds
INFO: Results and/or logs in: /var/lib/mock/viduxos-7-x86_64/result
ERROR: [Errno 2] No such file or directory: 'a2ps-4.14-23.el7.src.rpm'
Traceback (most recent call last):
  File "/usr/sbin/mock", line 306, in <module>
    def do_buildsrpm(config_opts, chroot, options, args):
  File "/usr/sbin/mock", line 630, in main
    do_rebuild(config_opts, chroot, args)
  File "<peak.util.decorators.rewrap wrapping __main__.do_rebuild at 0x00EB8B18>", line 3, in do_rebuild
    def do_rebuild(config_opts, chroot, srpms): return __decorated(config_opts, chroot, srpms)
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 70, in trace
    result = func(*args, **kw)
  File "/usr/sbin/mock", line 278, in do_rebuild
    chroot.build(srpm, timeout=config_opts['rpmbuild_timeout'], check=config_opts['check'])
  File "<peak.util.decorators.rewrap wrapping mockbuild.backend.build at 0x00EB8C80>", line 3, in build
    def build(self, srpm, timeout, check): return __decorated(self, srpm, timeout, check)
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 70, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.7/site-packages/mockbuild/backend.py", line 689, in build
    srpmChrootFilename = self._copySrpmIntoChroot(srpm)
  File "<peak.util.decorators.rewrap wrapping mockbuild.backend._copySrpmIntoChroot at 0x00EB9E60>", line 3, in _copySrpmIntoChroot
    def _copySrpmIntoChroot(self, srpm): return __decorated(self, srpm)
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 70, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.7/site-packages/mockbuild/backend.py", line 1092, in _copySrpmIntoChroot
    shutil.copy2(srpm, dest)
  File "/usr/lib64/python2.7/shutil.py", line 130, in copy2
    copyfile(src, dst)
  File "/usr/lib64/python2.7/shutil.py", line 82, in copyfile
    with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: 'a2ps-4.14-23.el7.src.rpm'

i even try to copy the src.rpm to the current working dir and just run:

mock -r viduxos-7-x86_64 --define "rhel 7" --define ".el7 1" --define "dist .el7" --rebuild a2ps-4.14-23.el7.src.rpm

and the same happened.

so finally find the root of the problem. when change something in site-defaults.cfg then mock should have to rebuild the cache. if i set it then the first run working ie when the system rebuild the cache. but the second run when start to use it than it gives the above error.

is there any other help how i can help...log debug version etc...?

it's also happened on with the latest 1.1.39 version on rhel7. i don't know how can it be possible that it's working for others!?

as i set 
config_opts['plugin_conf']['root_cache_enable'] = False
now everything works, but of course much more slower the before. so this's the worst workaround...

can i get some kind of debug version to find this bug? it's really slow down or build times. i'd be happy to test any kind of solution or tips!

is there any plan to be able to use root_cache_enable?

in 1.1.40 root_cache support still broken. this slow down build process so much! any chance to look into this problem?

Taking.

When site-default.cfg is updated then cache is invalidated. You should see:
INFO: enabled root cache
INFO: /etc/mock/site-defaults.cfg newer than root cache; cache will be rebuilt
INFO: enabled yum cache

That part is there since 2012.

Is it possible that your run mock from / So you are hitting 
https://bugzilla.redhat.com/show_bug.cgi?id=1160428#c12

Can you please try with mock-1.2.2 (in updates-testing right now).

the problem still exists!
so the root_cache still not working. which would be a very good thing to use!!! since currently all builds are very slow!

Start: build phase for virt-who-0.8-15.el7_0.src.rpm
Start: build setup for virt-who-0.8-15.el7_0.src.rpm
ERROR: Exception(virt-who-0.8-15.el7_0.src.rpm) Config(viduxos-7-x86_64) 0 minutes 11 seconds
INFO: Results and/or logs in: /var/lib/mock/viduxos-7-x86_64/result
INFO: unmounting tmpfs.
ERROR: [Errno 2] No such file or directory: 'virt-who-0.8-15.el7_0.src.rpm'
Traceback (most recent call last):
  File "/usr/sbin/mock", line 797, in <module>
    main()
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/usr/sbin/mock", line 628, in main
    run_command(options, args, config_opts, commands, buildroot, state)
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/usr/sbin/mock", line 703, in run_command
    do_rebuild(config_opts, commands, buildroot, args)
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/usr/sbin/mock", line 474, in do_rebuild
    post=createrepo_on_rpms, clean=clean)
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/usr/sbin/mock", line 427, in rebuild_generic
    ret = cmd(item)
  File "/usr/sbin/mock", line 462, in build
    check=config_opts['check'])
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.7/site-packages/mockbuild/backend.py", line 210, in build
    srpm = self.copy_srpm_into_chroot(srpm)
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 84, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.7/site-packages/mockbuild/backend.py", line 363, in copy_srpm_into_chroot
    shutil.copy2(srpm_path, dest)
  File "/usr/lib64/python2.7/shutil.py", line 130, in copy2
    copyfile(src, dst)
  File "/usr/lib64/python2.7/shutil.py", line 82, in copyfile
    with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: 'virt-who-0.8-15.el7_0.src.rpm'

I am using root_cache every day without problem.
Therefore I would say there is some problem related to your config.
Either please attach your config (both viduxos-7-x86_64.cfg and site-defaults.cfg) or reproduce that with default mock configs.

Please provide:
 * /etc/mock/site-defaults.cfg
 * your config viduxos-7-x86_64
so I can reproduce it.

Created attachment 1018430 [details]
site-defaults.cfg

Created attachment 1018431 [details]
viduxos-7-x86_64.cfg

and the reason of the bug is the 
config_opts['plugin_conf']['root_cache_enable'] = True
line in the config file. if i change it to False then it's working (although much more slower).

Looking at this BZ again. 

Well, it seem that the src.rpm exist at the start (otherwise there would be another error earlier), so it must be deleted once you start building.

I assume you did not delete it yourself. So it must be in some directory which mock wipe out.

Is there any chance that you run:

mock --rebuild /var/lib/mock/viduxos-7-x86_64/results/foo.src.rpm

?

Because that would explain that thing and will be duplicate of bug 1392137

not it's in an external directory ie. in our robot builder's home