Bug 1560460 - getFileStats fails on NFS domain in case or recursive symbolic link (e.g., using NetApp snapshots)
Summary: getFileStats fails on NFS domain in case or recursive symbolic link (e.g., us...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm
Version: 4.1.9
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ovirt-4.3.0
: 4.3.0
Assignee: Nir Soffer
QA Contact: Kevin Alon Goldblatt
URL:
Whiteboard:
Depends On: 1564515
Blocks: 1612904
TreeView+ depends on / blocked
 
Reported: 2018-03-26 08:44 UTC by Roman Hodain
Modified: 2020-08-03 15:32 UTC (History)
8 users (show)

Fixed In Version: v4.30.3
Doc Type: Bug Fix
Doc Text:
Previously, VDSM used stat() to implement islink() checks when using ioprocess to run commands. As a result, if a user or storage system created a recursive symbolic link inside the ISO storage domain, VDSM failed to report file information. In the current release, VDSM uses lstat() to implement islink() so it can report file information from recursive symbolic links.
Clone Of:
: 1612904 (view as bug list)
Environment:
Last Closed: 2019-05-08 12:36:02 UTC
oVirt Team: Storage
Target Upstream Version:
Embargoed:
lsvaty: testing_plan_complete-


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:1077 0 None None None 2019-05-08 12:36:24 UTC
oVirt gerrit 89901 0 master MERGED oop: Fix oop.os.path.islink() to use lstat() 2020-02-25 10:57:49 UTC
oVirt gerrit 93298 0 ovirt-4.2 MERGED oop: Fix oop.os.path.islink() to use lstat() 2020-02-25 10:57:49 UTC

Description Roman Hodain 2018-03-26 08:44:37 UTC
Description of problem:
NetApp snapshots feature may include symbolic links to NFS share which points to the same directory. This causes getFileStats to fail.

Example:
/rhev/data-center/mnt/<mountPoint>/.snapshot/daily.2018-02-20_0010 -> /rhev/data-center/mnt/<mountPoint>/<SD_UUID>/images/11111111-1111-1111-1111-111111111111/

Version-Release number of selected component (if applicable):
   python-ioprocess-0.16.1-1.el7ev.noarch
   ioprocess-0.16.1-1.el7ev.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Go to NFS ISO domain and create a recursive symbolic link.

     # pwd
       /rhev/data-center/mnt/10.34.1.195:_exports_rheviso01/c217bcfa-ffba-4982-b2c3-4ed418a0cd1d/images/11111111-1111-1111-1111-111111111111
     # ln -s ../ testLink

2. Try to get the list of isos in the WebAdmin


Actual results:
2018-03-26 10:08:03,596+0200 ERROR (jsonrpc/6) [storage.TaskManager.Task] (Task='70ba66b9-dd38-4042-9baa
-d390a8c44305') Unexpected error (task:872)
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 879, in _run
    return fn(*args, **kargs)
  File "<string>", line 2, in getFileStats
  File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method
    ret = func(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 2305, in getFileStats
    caseSensitive=caseSensitive)
  File "/usr/share/vdsm/storage/fileSD.py", line 496, in getFileList
    filesList = self.oop.simpleWalk(basedir)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 351, in simpleWalk
    files.extend(simpleWalk(ioproc, fullpath))
  File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 351, in simpleWalk
    files.extend(simpleWalk(ioproc, fullpath))
  File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 351, in simpleWalk
    files.extend(simpleWalk(ioproc, fullpath))
....
  File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 255, in isdir
    res = self._iop.stat(path)
  File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 484, in stat
    resdict = self._sendCommand("stat", {"path": path}, self.timeout)
  File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 461, in _sendCommand
    raise OSError(errcode, errstr)
OSError: [Errno 40] Too many levels of symbolic links
....
2018-03-26 10:08:03,600+0200 ERROR (jsonrpc/6) [storage.Dispatcher] FINISH getFileStats error=[Errno 40]
 Too many levels of symbolic links (dispatcher:85)
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/dispatcher.py", line 72, in wrapper
    result = ctask.prepare(func, *args, **kwargs)
  File "/usr/share/vdsm/storage/task.py", line 105, in wrapper
    return m(self, *a, **kw)
  File "/usr/share/vdsm/storage/task.py", line 1185, in prepare
    raise self.error
OSError: [Errno 40] Too many levels of symbolic links

Expected results:
The symbolic link is skipped.

Additional info:
ioprocess tries to skip the symlinks:

def simpleWalk(ioproc, path):
    files = []
    for f in ioproc.listdir(path):
        fullpath = os.path.join(path, f)
        osPath = _IOProcessOs(ioproc).path
        if osPath.isdir(fullpath) and not osPath.islink(fullpath):
            files.extend(simpleWalk(ioproc, fullpath))
        else:
            files.append(fullpath)

    return files

The problem is that the islink function uses "stat" function from ioprocess c library which does use a stat from sys/stat.h which returns information about the target not about the link itself. So lstat function should be used instead.

Comment 1 Nir Soffer 2018-03-26 18:36:01 UTC
Roman, can you share output of "ls -lhR" in /rhev/data-center/mnt/server:_path for a faulty server using these recursive symlinks?

Comment 5 Nir Soffer 2018-04-06 13:07:25 UTC
We don't have enough data to understand how a recursive link was created inside
a the iso domain images directory, but checking the source show that we indeed 
implement os.path.islink() in the wrong way in vdsm.

Comment 6 Nir Soffer 2018-04-06 13:32:34 UTC
Fixing this requires adding lstat() support in ioprocess, see bug 1564515.

Comment 7 RHV bug bot 2018-08-06 12:02:34 UTC
WARN: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{'rhevm-4.2.z': '?'}', ]

For more info please contact: rhv-devops: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{'rhevm-4.2.z': '?'}', ]

For more info please contact: rhv-devops

Comment 9 Elad 2018-08-21 12:39:35 UTC
Verify according to https://bugzilla.redhat.com/show_bug.cgi?id=1612904#c13

Comment 10 Kevin Alon Goldblatt 2018-10-17 16:20:27 UTC
Verified with the following code:
----------------------------------------
ovirt-engine-4.3.0-0.0.master.20181012165724.gitd25f971.el7.noarch
vdsm-4.30.0-640.git6fd8327.el7.x86_64

Verified with the following scenario:
----------------------------------------
Steps to Reproduce:
1. Go to NFS ISO domain and create a recursive symbolic link.

     # pwd
       /rhev/data-center/mnt/10.34.1.195:_exports_rheviso01/c217bcfa-ffba-4982-b2c3-4ed418a0cd1d/images/11111111-1111-1111-1111-111111111111
     # ln -s ../ testLink

2. Try to get the list of isos in the WebAdmin >>>>> Works fine and no errors are reported

Comment 13 errata-xmlrpc 2019-05-08 12:36:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1077

Comment 14 Daniel Gur 2019-08-28 13:11:30 UTC
sync2jira

Comment 15 Daniel Gur 2019-08-28 13:15:41 UTC
sync2jira


Note You need to log in before you can comment on or make changes to this bug.