Created attachment 926697 [details] vdsm.log Split from: https://bugzilla.redhat.com/show_bug.cgi?id=1114499#c6 Images on ISO domain still not visible ~~~ ...snip... Thread-127::ERROR::2014-08-07 14:48:03,222::task::866::Storage.TaskManager.Task::(_setError) Task=`adb45a68-d297-46d0-89fe-c1da4e9cdb45`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 873, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 45, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 2302, in getFileStats caseSensitive=caseSensitive) File "/usr/share/vdsm/storage/fileSD.py", line 281, in getFileList st = self.oop.os.stat(entry) File "/usr/share/vdsm/storage/outOfProcess.py", line 241, in stat return self._iop.stat(path) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 367, in stat resdict = self._sendCommand("stat", {"path": path}, self.timeout) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 344, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory ...snip... Thread-127::ERROR::2014-08-07 14:48:03,225::dispatcher::79::Storage.Dispatcher::(wrapper) [Errno 2] No such file or directory Traceback (most recent call last): File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/share/vdsm/storage/task.py", line 103, in wrapper return m(self, *a, **kw) File "/usr/share/vdsm/storage/task.py", line 1179, in prepare raise self.error OSError: [Errno 2] No such file or directory ~~~ Version-Release number of selected component (if applicable): vdsm-jsonrpc-4.16.1-6.gita4a4614.el6.noarch vdsm-4.16.1-6.gita4a4614.el6.x86_64 vdsm-python-zombiereaper-4.16.1-6.gita4a4614.el6.noarch vdsm-xmlrpc-4.16.1-6.gita4a4614.el6.noarch vdsm-yajsonrpc-4.16.1-6.gita4a4614.el6.noarch vdsm-cli-4.16.1-6.gita4a4614.el6.noarch vdsm-python-4.16.1-6.gita4a4614.el6.x86_64 How reproducible: 100% Steps to Reproduce: 1. add iso storage 2. check images in Admin Portal (click refresh button) 3. Actual results: no list Expected results: list of images Additional info: Original discussion at https://bugzilla.redhat.com/show_bug.cgi?id=1114499 Workaround https://bugzilla.redhat.com/show_bug.cgi?id=1114499#c1 doesn't work for me anymore.
I've looked into the issue, the cause here is that on the iso domain there's one iso which is actually a symbolic link which causes us to fail when executing the stat. Moving this bug to infra to decide how to proceed with it, I also suggest to add further logging in the infra so that it'll be easier to inspect on what exactly we fail.
[root@str-02 11111111-1111-1111-1111-111111111111]# stat ovirt-node-iso-3.5.0.ovirt35.20140805.0.el6.iso File: `ovirt-node-iso-3.5.0.ovirt35.20140805.0.el6.iso' -> `/home/httpd/repo-tlv-builds/ovirt-3.5-pre/iso/ovirt-node-iso-3.5.0.ovirt35.20140805.0.el6.iso' Size: 93 Blocks: 8 IO Block: 4096 symbolic link Device: 803h/2051d Inode: 6029494 Links: 1 Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2014-08-27 15:31:00.779344025 +0200 Modify: 2014-08-07 10:54:34.525343959 +0200 Change: 2014-08-07 10:54:34.525343959 +0200 after - mv ovirt-node-iso-3.5.0.ovirt35.20140805.0.el6.iso ovirt-node-iso-3.5.0.ovirt35.20140805.0.el6.isotmp it succeeds.
(i'm not fixing anything on the verb assuming that "stat" failure is never expected).
The behavior of what needs to be seen in iso domain is purly storage related. both in terms of behavior and in terms of code. Moving back to storage.
Jiri, please try to reproduce it without ioprocess to see if that issue existed also before.
Please ignore my last comment, I'll elaborate more here. The issue seems to be that the linked file doesn't exist, and that our stat call tries to reach to the linked file and not to return the stat of the link. Jiri, if you can please verify that on your env (as i don't have the exact scenario) and also let's verify that this was always the behavior. thanks, Liron.
Created attachment 934091 [details] engine.log & vdsm.log Yes, the problem is symlink. But in my case I can't get ISO list either symlink works or is broken and it points outside of the directory with ISOs. If symlink works and it is in same directory as all ISOs, it works OK. If it does not again no ISO list.
(In reply to Jiri Belka from comment #0) [...] > Expected results: > list of images Depending on the symlink'd iso location, the link target is likely not even mounted on the host where the iso domain is mounted so some failure would be expected. We might be able to show all images that are stat-able, though the argument could be made that it's better to not silently ignore an error such as the domain containing an unreachable image. What do you think?
I don't know what would be right solution but I was recently getting target of a symlink and I was told that this introduces race. So take it into account or close this BZ.
ISO domain management requires being able to stat the files, which can't be done reliably on a symlink. The effort to change this behavior is not worth the reward or the risk. Closing this as WONTFIX
Andrew, this limitation should probably be documented
Hi Tal, Thank you for the needinfo request. Just to confirm - should we inform users to *not* use symlinks for ISO files, or should we document the behavior that symlinks will work, but only when the object they are pointing to actually exists? Kind regards, Andrew
(In reply to Andrew Dahms from comment #12) > Hi Tal, > > Thank you for the needinfo request. > > Just to confirm - should we inform users to *not* use symlinks for ISO > files, or should we document the behavior that symlinks will work, but only > when the object they are pointing to actually exists? > > Kind regards, > > Andrew Hi Andrew, Not Tal, but I'll answer :) The former, no symlinks. Not only are they error-prone in this case (they work in a very few specific scenarios), but also we're not planning to manage symlink'd images from Image Upload and having them on the domain may cause unexpected failures. Thanks, Greg
*** Bug 1280225 has been marked as a duplicate of this bug. ***