Bug 1612904

Summary: [downstream clone - 4.2.6] getFileStats fails on NFS domain in case or recursive symbolic link (e.g., using NetApp snapshots)
Product: Red Hat Enterprise Virtualization Manager Reporter: RHV bug bot <rhv-bugzilla-bot>
Component: vdsmAssignee: Nir Soffer <nsoffer>
Status: CLOSED ERRATA QA Contact: Kevin Alon Goldblatt <kgoldbla>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.1.9CC: dfodor, eheftman, lsurette, michal.skrivanek, nsoffer, rhodain, srevivo, tnisan, ycui, ylavi
Target Milestone: ovirt-4.2.6Keywords: ZStream
Target Release: ---Flags: lsvaty: testing_plan_complete-
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: v4.20.37 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1560460 Environment:
Last Closed: 2018-09-04 13:43:24 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1560460    
Bug Blocks:    
Attachments:
Description Flags
engine and vdsm logs none

Description RHV bug bot 2018-08-06 13:47:11 UTC
+++ This bug is a downstream clone. The original bug is: +++
+++   bug 1560460 +++
======================================================================

Description of problem:
NetApp snapshots feature may include symbolic links to NFS share which points to the same directory. This causes getFileStats to fail.

Example:
/rhev/data-center/mnt/<mountPoint>/.snapshot/daily.2018-02-20_0010 -> /rhev/data-center/mnt/<mountPoint>/<SD_UUID>/images/11111111-1111-1111-1111-111111111111/

Version-Release number of selected component (if applicable):
   python-ioprocess-0.16.1-1.el7ev.noarch
   ioprocess-0.16.1-1.el7ev.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Go to NFS ISO domain and create a recursive symbolic link.

     # pwd
       /rhev/data-center/mnt/10.34.1.195:_exports_rheviso01/c217bcfa-ffba-4982-b2c3-4ed418a0cd1d/images/11111111-1111-1111-1111-111111111111
     # ln -s ../ testLink

2. Try to get the list of isos in the WebAdmin


Actual results:
2018-03-26 10:08:03,596+0200 ERROR (jsonrpc/6) [storage.TaskManager.Task] (Task='70ba66b9-dd38-4042-9baa
-d390a8c44305') Unexpected error (task:872)
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 879, in _run
    return fn(*args, **kargs)
  File "<string>", line 2, in getFileStats
  File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method
    ret = func(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 2305, in getFileStats
    caseSensitive=caseSensitive)
  File "/usr/share/vdsm/storage/fileSD.py", line 496, in getFileList
    filesList = self.oop.simpleWalk(basedir)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 351, in simpleWalk
    files.extend(simpleWalk(ioproc, fullpath))
  File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 351, in simpleWalk
    files.extend(simpleWalk(ioproc, fullpath))
  File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 351, in simpleWalk
    files.extend(simpleWalk(ioproc, fullpath))
....
  File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 255, in isdir
    res = self._iop.stat(path)
  File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 484, in stat
    resdict = self._sendCommand("stat", {"path": path}, self.timeout)
  File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 461, in _sendCommand
    raise OSError(errcode, errstr)
OSError: [Errno 40] Too many levels of symbolic links
....
2018-03-26 10:08:03,600+0200 ERROR (jsonrpc/6) [storage.Dispatcher] FINISH getFileStats error=[Errno 40]
 Too many levels of symbolic links (dispatcher:85)
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/dispatcher.py", line 72, in wrapper
    result = ctask.prepare(func, *args, **kwargs)
  File "/usr/share/vdsm/storage/task.py", line 105, in wrapper
    return m(self, *a, **kw)
  File "/usr/share/vdsm/storage/task.py", line 1185, in prepare
    raise self.error
OSError: [Errno 40] Too many levels of symbolic links

Expected results:
The symbolic link is skipped.

Additional info:
ioprocess tries to skip the symlinks:

def simpleWalk(ioproc, path):
    files = []
    for f in ioproc.listdir(path):
        fullpath = os.path.join(path, f)
        osPath = _IOProcessOs(ioproc).path
        if osPath.isdir(fullpath) and not osPath.islink(fullpath):
            files.extend(simpleWalk(ioproc, fullpath))
        else:
            files.append(fullpath)

    return files

The problem is that the islink function uses "stat" function from ioprocess c library which does use a stat from sys/stat.h which returns information about the target not about the link itself. So lstat function should be used instead.

(Originally by Roman Hodain)

Comment 1 RHV bug bot 2018-08-06 13:47:16 UTC
Roman, can you share output of "ls -lhR" in /rhev/data-center/mnt/server:_path for a faulty server using these recursive symlinks?

(Originally by Nir Soffer)

Comment 5 RHV bug bot 2018-08-06 13:47:31 UTC
We don't have enough data to understand how a recursive link was created inside
a the iso domain images directory, but checking the source show that we indeed 
implement os.path.islink() in the wrong way in vdsm.

(Originally by Nir Soffer)

Comment 6 RHV bug bot 2018-08-06 13:47:35 UTC
Fixing this requires adding lstat() support in ioprocess, see bug 1564515.

(Originally by Nir Soffer)

Comment 7 RHV bug bot 2018-08-06 13:47:39 UTC
WARN: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{'rhevm-4.2.z': '?'}', ]

For more info please contact: rhv-devops: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{'rhevm-4.2.z': '?'}', ]

For more info please contact: rhv-devops

(Originally by rhv-bugzilla-bot)

Comment 11 Kevin Alon Goldblatt 2018-08-09 09:41:56 UTC
Created attachment 1474619 [details]
engine and vdsm logs

Adding logs

Comment 13 Kevin Alon Goldblatt 2018-08-09 11:59:03 UTC
Based on Comment 12 and result from comment 8 moving to VERIFIED!

Comment 15 errata-xmlrpc 2018-09-04 13:43:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2624

Comment 16 Daniel Gur 2019-08-28 13:14:22 UTC
sync2jira

Comment 17 Daniel Gur 2019-08-28 13:19:24 UTC
sync2jira