Bug 1612904 - [downstream clone - 4.2.6] getFileStats fails on NFS domain in case or recursive symbolic link (e.g., using NetApp snapshots)
Summary: [downstream clone - 4.2.6] getFileStats fails on NFS domain in case or recurs...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm
Version: 4.1.9
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ovirt-4.2.6
: ---
Assignee: Nir Soffer
QA Contact: Kevin Alon Goldblatt
URL:
Whiteboard:
Depends On: 1560460
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-08-06 13:47 UTC by RHV bug bot
Modified: 2020-08-03 15:29 UTC (History)
10 users (show)

Fixed In Version: v4.20.37
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1560460
Environment:
Last Closed: 2018-09-04 13:43:24 UTC
oVirt Team: Storage
Target Upstream Version:
Embargoed:
lsvaty: testing_plan_complete-


Attachments (Terms of Use)
engine and vdsm logs (618.29 KB, application/x-gzip)
2018-08-09 09:41 UTC, Kevin Alon Goldblatt
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2018:2624 0 None None None 2018-09-04 13:43:46 UTC
oVirt gerrit 89901 0 master MERGED oop: Fix oop.os.path.islink() to use lstat() 2018-08-06 13:47:49 UTC
oVirt gerrit 93298 0 ovirt-4.2 MERGED oop: Fix oop.os.path.islink() to use lstat() 2018-08-06 13:47:49 UTC

Description RHV bug bot 2018-08-06 13:47:11 UTC
+++ This bug is a downstream clone. The original bug is: +++
+++   bug 1560460 +++
======================================================================

Description of problem:
NetApp snapshots feature may include symbolic links to NFS share which points to the same directory. This causes getFileStats to fail.

Example:
/rhev/data-center/mnt/<mountPoint>/.snapshot/daily.2018-02-20_0010 -> /rhev/data-center/mnt/<mountPoint>/<SD_UUID>/images/11111111-1111-1111-1111-111111111111/

Version-Release number of selected component (if applicable):
   python-ioprocess-0.16.1-1.el7ev.noarch
   ioprocess-0.16.1-1.el7ev.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Go to NFS ISO domain and create a recursive symbolic link.

     # pwd
       /rhev/data-center/mnt/10.34.1.195:_exports_rheviso01/c217bcfa-ffba-4982-b2c3-4ed418a0cd1d/images/11111111-1111-1111-1111-111111111111
     # ln -s ../ testLink

2. Try to get the list of isos in the WebAdmin


Actual results:
2018-03-26 10:08:03,596+0200 ERROR (jsonrpc/6) [storage.TaskManager.Task] (Task='70ba66b9-dd38-4042-9baa
-d390a8c44305') Unexpected error (task:872)
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 879, in _run
    return fn(*args, **kargs)
  File "<string>", line 2, in getFileStats
  File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method
    ret = func(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 2305, in getFileStats
    caseSensitive=caseSensitive)
  File "/usr/share/vdsm/storage/fileSD.py", line 496, in getFileList
    filesList = self.oop.simpleWalk(basedir)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 351, in simpleWalk
    files.extend(simpleWalk(ioproc, fullpath))
  File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 351, in simpleWalk
    files.extend(simpleWalk(ioproc, fullpath))
  File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 351, in simpleWalk
    files.extend(simpleWalk(ioproc, fullpath))
....
  File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 255, in isdir
    res = self._iop.stat(path)
  File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 484, in stat
    resdict = self._sendCommand("stat", {"path": path}, self.timeout)
  File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 461, in _sendCommand
    raise OSError(errcode, errstr)
OSError: [Errno 40] Too many levels of symbolic links
....
2018-03-26 10:08:03,600+0200 ERROR (jsonrpc/6) [storage.Dispatcher] FINISH getFileStats error=[Errno 40]
 Too many levels of symbolic links (dispatcher:85)
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/dispatcher.py", line 72, in wrapper
    result = ctask.prepare(func, *args, **kwargs)
  File "/usr/share/vdsm/storage/task.py", line 105, in wrapper
    return m(self, *a, **kw)
  File "/usr/share/vdsm/storage/task.py", line 1185, in prepare
    raise self.error
OSError: [Errno 40] Too many levels of symbolic links

Expected results:
The symbolic link is skipped.

Additional info:
ioprocess tries to skip the symlinks:

def simpleWalk(ioproc, path):
    files = []
    for f in ioproc.listdir(path):
        fullpath = os.path.join(path, f)
        osPath = _IOProcessOs(ioproc).path
        if osPath.isdir(fullpath) and not osPath.islink(fullpath):
            files.extend(simpleWalk(ioproc, fullpath))
        else:
            files.append(fullpath)

    return files

The problem is that the islink function uses "stat" function from ioprocess c library which does use a stat from sys/stat.h which returns information about the target not about the link itself. So lstat function should be used instead.

(Originally by Roman Hodain)

Comment 1 RHV bug bot 2018-08-06 13:47:16 UTC
Roman, can you share output of "ls -lhR" in /rhev/data-center/mnt/server:_path for a faulty server using these recursive symlinks?

(Originally by Nir Soffer)

Comment 5 RHV bug bot 2018-08-06 13:47:31 UTC
We don't have enough data to understand how a recursive link was created inside
a the iso domain images directory, but checking the source show that we indeed 
implement os.path.islink() in the wrong way in vdsm.

(Originally by Nir Soffer)

Comment 6 RHV bug bot 2018-08-06 13:47:35 UTC
Fixing this requires adding lstat() support in ioprocess, see bug 1564515.

(Originally by Nir Soffer)

Comment 7 RHV bug bot 2018-08-06 13:47:39 UTC
WARN: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{'rhevm-4.2.z': '?'}', ]

For more info please contact: rhv-devops: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{'rhevm-4.2.z': '?'}', ]

For more info please contact: rhv-devops

(Originally by rhv-bugzilla-bot)

Comment 11 Kevin Alon Goldblatt 2018-08-09 09:41:56 UTC
Created attachment 1474619 [details]
engine and vdsm logs

Adding logs

Comment 13 Kevin Alon Goldblatt 2018-08-09 11:59:03 UTC
Based on Comment 12 and result from comment 8 moving to VERIFIED!

Comment 15 errata-xmlrpc 2018-09-04 13:43:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2624

Comment 16 Daniel Gur 2019-08-28 13:14:22 UTC
sync2jira

Comment 17 Daniel Gur 2019-08-28 13:19:24 UTC
sync2jira


Note You need to log in before you can comment on or make changes to this bug.