Bug 1130024

Summary: Can't see images on ISO domain via symlink
Product: [Retired] oVirt Reporter: Jiri Belka <jbelka>
Component: ovirt-engine-coreAssignee: Greg Padgett <gpadgett>
Status: CLOSED WONTFIX QA Contact: Aharon Canan <acanan>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.5CC: adahms, amureini, bugs, ecohen, jbelka, laravot, michal.skrivanek, rbalakri, stirabos, tnisan, yeylon, ylavi
Target Milestone: m1   
Target Release: 3.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: storage
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-08-13 10:54:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
vdsm.log
none
engine.log & vdsm.log none

Description Jiri Belka 2014-08-14 08:08:42 UTC 
Created attachment 926697 [details]
vdsm.log

Split from: https://bugzilla.redhat.com/show_bug.cgi?id=1114499#c6

Images on ISO domain still not visible

~~~
...snip...
Thread-127::ERROR::2014-08-07 14:48:03,222::task::866::Storage.TaskManager.Task::(_setError) Task=`adb45a68-d297-46d0-89fe-c1da4e9cdb45`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 2302, in getFileStats
    caseSensitive=caseSensitive)
  File "/usr/share/vdsm/storage/fileSD.py", line 281, in getFileList
    st = self.oop.os.stat(entry)
  File "/usr/share/vdsm/storage/outOfProcess.py", line 241, in stat
    return self._iop.stat(path)
  File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 367, in stat
    resdict = self._sendCommand("stat", {"path": path}, self.timeout)
  File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 344, in _sendCommand
    raise OSError(errcode, errstr)
OSError: [Errno 2] No such file or directory
...snip...
Thread-127::ERROR::2014-08-07 14:48:03,225::dispatcher::79::Storage.Dispatcher::(wrapper) [Errno 2] No such file or directory
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper
    result = ctask.prepare(func, *args, **kwargs)
  File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
    return m(self, *a, **kw)
  File "/usr/share/vdsm/storage/task.py", line 1179, in prepare
    raise self.error
OSError: [Errno 2] No such file or directory
~~~

Version-Release number of selected component (if applicable):
vdsm-jsonrpc-4.16.1-6.gita4a4614.el6.noarch
vdsm-4.16.1-6.gita4a4614.el6.x86_64
vdsm-python-zombiereaper-4.16.1-6.gita4a4614.el6.noarch
vdsm-xmlrpc-4.16.1-6.gita4a4614.el6.noarch
vdsm-yajsonrpc-4.16.1-6.gita4a4614.el6.noarch
vdsm-cli-4.16.1-6.gita4a4614.el6.noarch
vdsm-python-4.16.1-6.gita4a4614.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1. add iso storage
2. check images in Admin Portal (click refresh button)
3.

Actual results:
no list

Expected results:
list of images

Additional info:
Original discussion at https://bugzilla.redhat.com/show_bug.cgi?id=1114499
Workaround https://bugzilla.redhat.com/show_bug.cgi?id=1114499#c1 doesn't work for me anymore.
Comment 1 Liron Aravot 2014-08-27 13:54:36 UTC 
I've looked into the issue, the cause here is that on the iso domain there's one iso which is actually a symbolic link which causes us to fail when executing the stat.

Moving this bug to infra to decide how to proceed with it, I also suggest to add further logging in the infra so that it'll be easier to inspect on what exactly we fail.
Comment 2 Liron Aravot 2014-08-27 13:56:18 UTC 
[root@str-02 11111111-1111-1111-1111-111111111111]# stat ovirt-node-iso-3.5.0.ovirt35.20140805.0.el6.iso
  File: `ovirt-node-iso-3.5.0.ovirt35.20140805.0.el6.iso' -> `/home/httpd/repo-tlv-builds/ovirt-3.5-pre/iso/ovirt-node-iso-3.5.0.ovirt35.20140805.0.el6.iso'
  Size: 93        	Blocks: 8          IO Block: 4096   symbolic link
Device: 803h/2051d	Inode: 6029494     Links: 1
Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2014-08-27 15:31:00.779344025 +0200
Modify: 2014-08-07 10:54:34.525343959 +0200
Change: 2014-08-07 10:54:34.525343959 +0200


after -
mv ovirt-node-iso-3.5.0.ovirt35.20140805.0.el6.iso ovirt-node-iso-3.5.0.ovirt35.20140805.0.el6.isotmp

it succeeds.
Comment 3 Liron Aravot 2014-08-28 07:51:33 UTC 
(i'm not fixing anything on the verb assuming that "stat" failure is never expected).
Comment 4 Barak 2014-09-01 09:48:22 UTC 
The behavior of what needs to be seen in iso domain is purly storage related.
both in terms of behavior and in terms of code.

Moving back to storage.
Comment 5 Liron Aravot 2014-09-01 11:34:17 UTC 
Jiri, please try to reproduce it without ioprocess to see if that issue existed also before.
Comment 6 Liron Aravot 2014-09-01 12:27:46 UTC 
Please ignore my last comment, I'll elaborate more here.
The issue seems to be that the linked file doesn't exist, and that our stat call tries to reach to the linked file and not to return the stat of the link.

Jiri, if you can please verify that on your env (as i don't have the exact scenario) and also let's verify that this was always the behavior.

thanks,
Liron.
Comment 7 Jiri Belka 2014-09-03 13:24:46 UTC 
Created attachment 934091 [details]
engine.log & vdsm.log

Yes, the problem is symlink.

But in my case I can't get ISO list either symlink works or is broken and it points outside of the directory with ISOs.

If symlink works and it is in same directory as all ISOs, it works OK. If it does not again no ISO list.
Comment 8 Greg Padgett 2015-08-03 23:15:23 UTC 
(In reply to Jiri Belka from comment #0)
[...]
> Expected results:
> list of images

Depending on the symlink'd iso location, the link target is likely not even mounted on the host where the iso domain is mounted so some failure would be expected.  We might be able to show all images that are stat-able, though the argument could be made that it's better to not silently ignore an error such as the domain containing an unreachable image.

What do you think?
Comment 9 Jiri Belka 2015-08-11 08:56:10 UTC 
I don't know what would be right solution but I was recently getting target of a symlink and I was told that this introduces race. So take it into account or close this BZ.
Comment 10 Tal Nisan 2015-08-13 10:54:46 UTC 
ISO domain management requires being able to stat the files, which can't be done reliably on a symlink. The effort to change this behavior is not worth the reward or the risk. Closing this as WONTFIX
Comment 11 Tal Nisan 2015-08-13 10:56:37 UTC 
Andrew, this limitation should probably be documented
Comment 12 Andrew Dahms 2015-08-14 04:10:24 UTC 
Hi Tal,

Thank you for the needinfo request.

Just to confirm - should we inform users to *not* use symlinks for ISO files, or should we document the behavior that symlinks will work, but only when the object they are pointing to actually exists?

Kind regards,

Andrew
Comment 13 Greg Padgett 2015-08-17 18:34:18 UTC 
(In reply to Andrew Dahms from comment #12)
> Hi Tal,
> 
> Thank you for the needinfo request.
> 
> Just to confirm - should we inform users to *not* use symlinks for ISO
> files, or should we document the behavior that symlinks will work, but only
> when the object they are pointing to actually exists?
> 
> Kind regards,
> 
> Andrew

Hi Andrew,

Not Tal, but I'll answer :)  The former, no symlinks.  Not only are they error-prone in this case (they work in a very few specific scenarios), but also we're not planning to manage symlink'd images from Image Upload and having them on the domain may cause unexpected failures.

Thanks,
Greg
Comment 14 Allon Mureinik 2015-11-19 12:35:41 UTC 
*** Bug 1280225 has been marked as a duplicate of this bug. ***