Description of problem: See summary Version-Release number of selected component (if applicable): Current master nightly snapshot How reproducible: Not sure Steps to Reproduce: 1. install and setup engine with local iso domain 2. add a host 3. add an iso file to the iso domain 4. refresh Actual results: Fails, vdsm.log has: Thread-61::ERROR::2014-06-30 11:27:57,441::task::866::Storage.TaskManager.Task::(_setError) Task=`d9c6310b-0b5f-4634-a380-13ea607584ea`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 873, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 45, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 2324, in getIsoList caseSensitive=False) File "/usr/share/vdsm/storage/fileSD.py", line 285, in getFileList self.oop.fileUtils.validateQemuReadable(entry) AttributeError: '_IOProcessFileUtils' object has no attribute 'validateQemuReadable' Expected results: Succeeds Additional info:
Workaround, following Dan's suggestion: Edit /etc/vdsm/vdsm.conf to have: [irs] oop_impl = rfh and restart vdsm.
*** Bug 1116561 has been marked as a duplicate of this bug. ***
*** Bug 1114057 has been marked as a duplicate of this bug. ***
*** Bug 1114052 has been marked as a duplicate of this bug. ***
*** Bug 1117688 has been marked as a duplicate of this bug. ***
Created attachment 924900 [details] vdsm.log Images on ISO domain still not visible, please check newly added vdsm.log. If different issue please let me know, I'll file new BZ and make this one depend on it. ~~~ ...snip... Thread-127::ERROR::2014-08-07 14:48:03,222::task::866::Storage.TaskManager.Task::(_setError) Task=`adb45a68-d297-46d0-89fe-c1da4e9cdb45`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 873, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 45, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 2302, in getFileStats caseSensitive=caseSensitive) File "/usr/share/vdsm/storage/fileSD.py", line 281, in getFileList st = self.oop.os.stat(entry) File "/usr/share/vdsm/storage/outOfProcess.py", line 241, in stat return self._iop.stat(path) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 367, in stat resdict = self._sendCommand("stat", {"path": path}, self.timeout) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 344, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory ...snip... Thread-127::ERROR::2014-08-07 14:48:03,225::dispatcher::79::Storage.Dispatcher::(wrapper) [Errno 2] No such file or directory Traceback (most recent call last): File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/share/vdsm/storage/task.py", line 103, in wrapper return m(self, *a, **kw) File "/usr/share/vdsm/storage/task.py", line 1179, in prepare raise self.error OSError: [Errno 2] No such file or directory ~~~
# rpm -qa vdsm\* vdsm-jsonrpc-4.16.1-6.gita4a4614.el6.noarch vdsm-4.16.1-6.gita4a4614.el6.x86_64 vdsm-python-zombiereaper-4.16.1-6.gita4a4614.el6.noarch vdsm-xmlrpc-4.16.1-6.gita4a4614.el6.noarch vdsm-yajsonrpc-4.16.1-6.gita4a4614.el6.noarch vdsm-cli-4.16.1-6.gita4a4614.el6.noarch vdsm-python-4.16.1-6.gita4a4614.el6.x86_64
Yes, Jiri. It is a different issue. You found a race in getFileList: it globs for the filenames, then calls stat() and validateQemuReadable(), assuming the files would stay there forever. Could it be that you (or someone else) were removing files from the ISO domain while verifying this bug? Is this traceback reproducible? In any case, please open a new BZ.
Or could it be that the ISO domain was actually empty? Thread-127::DEBUG::2014-08-07 14:17:28,088::task::885::Storage.TaskManager.Task::(_run) Task=`95de7f78-5405-420b-b231-c62ff9be9c21`::Task._run: 95de7f78-5405-420b-b231-c62ff9be9c21 ('0c78b4d6-ba00-4d3e-9f9f-65c7d5899d71', '*.iso', False) {} failed - stopping task
Jiri, It does not fail on the original problem(validateQemuReadable but on stat). It's a different issue, and we might need a separate bug for this. What ioprocess version did you use to verify this? It might be that you need a newer version
Workaround in #1 doesn't work for me anymore. New BZ1130024 for issue in #6. Question from #8, #9 - no iso domain was ok, not empty and nobody was touching it iirc.
Moving back to ON_QA. Workaround was to switch to use rfh for outOfProcess, instead of ioprocess. It's a different issue that is unrelated to ioprocess and needs to be investigated. This bug needs to be verified in a different, clean environment because it is unrelated to the new bug, or depend on the new bug.
Jiri, could you verify it on a clean environment because of the urgency? thanks
ok, no traceback related to issue as described in #8 and #10, tested on clean rhel6 host. although there's still stat() issue as described in #10. vdsm-python-zombiereaper-4.16.1-6.gita4a4614.el6.noarch vdsm-jsonrpc-4.16.1-6.gita4a4614.el6.noarch python-ioprocess-0.5.0-1.el6.noarch vdsm-xmlrpc-4.16.1-6.gita4a4614.el6.noarch vdsm-yajsonrpc-4.16.1-6.gita4a4614.el6.noarch vdsm-4.16.1-6.gita4a4614.el6.x86_64 vdsm-python-4.16.1-6.gita4a4614.el6.x86_64 vdsm-cli-4.16.1-6.gita4a4614.el6.noarch
oVirt 3.5 has been released and should include the fix for this issue.