Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 1612904 - [downstream clone - 4.2.6] getFileStats fails on NFS domain in case or recursive symbolic link (e.g., using NetApp snapshots)
[downstream clone - 4.2.6] getFileStats fails on NFS domain in case or recurs...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm (Show other bugs)
4.1.9
Unspecified Unspecified
medium Severity medium
: ovirt-4.2.6
: ---
Assigned To: Nir Soffer
Kevin Alon Goldblatt
: ZStream
Depends On: 1560460
Blocks:
  Show dependency treegraph
 
Reported: 2018-08-06 09:47 EDT by RHV Bugzilla Automation and Verification Bot
Modified: 2018-09-04 09:43 EDT (History)
11 users (show)

See Also:
Fixed In Version: v4.20.37
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1560460
Environment:
Last Closed: 2018-09-04 09:43:24 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Storage
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
engine and vdsm logs (618.29 KB, application/x-gzip)
2018-08-09 05:41 EDT, Kevin Alon Goldblatt
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 89901 master MERGED oop: Fix oop.os.path.islink() to use lstat() 2018-08-06 09:47 EDT
oVirt gerrit 93298 ovirt-4.2 MERGED oop: Fix oop.os.path.islink() to use lstat() 2018-08-06 09:47 EDT
Red Hat Product Errata RHEA-2018:2624 None None None 2018-09-04 09:43 EDT

  None (edit)
Description RHV Bugzilla Automation and Verification Bot 2018-08-06 09:47:11 EDT
+++ This bug is a downstream clone. The original bug is: +++
+++   bug 1560460 +++
======================================================================

Description of problem:
NetApp snapshots feature may include symbolic links to NFS share which points to the same directory. This causes getFileStats to fail.

Example:
/rhev/data-center/mnt/<mountPoint>/.snapshot/daily.2018-02-20_0010 -> /rhev/data-center/mnt/<mountPoint>/<SD_UUID>/images/11111111-1111-1111-1111-111111111111/

Version-Release number of selected component (if applicable):
   python-ioprocess-0.16.1-1.el7ev.noarch
   ioprocess-0.16.1-1.el7ev.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Go to NFS ISO domain and create a recursive symbolic link.

     # pwd
       /rhev/data-center/mnt/10.34.1.195:_exports_rheviso01/c217bcfa-ffba-4982-b2c3-4ed418a0cd1d/images/11111111-1111-1111-1111-111111111111
     # ln -s ../ testLink

2. Try to get the list of isos in the WebAdmin


Actual results:
2018-03-26 10:08:03,596+0200 ERROR (jsonrpc/6) [storage.TaskManager.Task] (Task='70ba66b9-dd38-4042-9baa
-d390a8c44305') Unexpected error (task:872)
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 879, in _run
    return fn(*args, **kargs)
  File "<string>", line 2, in getFileStats
  File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method
    ret = func(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 2305, in getFileStats
    caseSensitive=caseSensitive)
  File "/usr/share/vdsm/storage/fileSD.py", line 496, in getFileList
    filesList = self.oop.simpleWalk(basedir)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 351, in simpleWalk
    files.extend(simpleWalk(ioproc, fullpath))
  File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 351, in simpleWalk
    files.extend(simpleWalk(ioproc, fullpath))
  File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 351, in simpleWalk
    files.extend(simpleWalk(ioproc, fullpath))
....
  File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 255, in isdir
    res = self._iop.stat(path)
  File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 484, in stat
    resdict = self._sendCommand("stat", {"path": path}, self.timeout)
  File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 461, in _sendCommand
    raise OSError(errcode, errstr)
OSError: [Errno 40] Too many levels of symbolic links
....
2018-03-26 10:08:03,600+0200 ERROR (jsonrpc/6) [storage.Dispatcher] FINISH getFileStats error=[Errno 40]
 Too many levels of symbolic links (dispatcher:85)
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/dispatcher.py", line 72, in wrapper
    result = ctask.prepare(func, *args, **kwargs)
  File "/usr/share/vdsm/storage/task.py", line 105, in wrapper
    return m(self, *a, **kw)
  File "/usr/share/vdsm/storage/task.py", line 1185, in prepare
    raise self.error
OSError: [Errno 40] Too many levels of symbolic links

Expected results:
The symbolic link is skipped.

Additional info:
ioprocess tries to skip the symlinks:

def simpleWalk(ioproc, path):
    files = []
    for f in ioproc.listdir(path):
        fullpath = os.path.join(path, f)
        osPath = _IOProcessOs(ioproc).path
        if osPath.isdir(fullpath) and not osPath.islink(fullpath):
            files.extend(simpleWalk(ioproc, fullpath))
        else:
            files.append(fullpath)

    return files

The problem is that the islink function uses "stat" function from ioprocess c library which does use a stat from sys/stat.h which returns information about the target not about the link itself. So lstat function should be used instead.

(Originally by Roman Hodain)
Comment 1 RHV Bugzilla Automation and Verification Bot 2018-08-06 09:47:16 EDT
Roman, can you share output of "ls -lhR" in /rhev/data-center/mnt/server:_path for a faulty server using these recursive symlinks?

(Originally by Nir Soffer)
Comment 5 RHV Bugzilla Automation and Verification Bot 2018-08-06 09:47:31 EDT
We don't have enough data to understand how a recursive link was created inside
a the iso domain images directory, but checking the source show that we indeed 
implement os.path.islink() in the wrong way in vdsm.

(Originally by Nir Soffer)
Comment 6 RHV Bugzilla Automation and Verification Bot 2018-08-06 09:47:35 EDT
Fixing this requires adding lstat() support in ioprocess, see bug 1564515.

(Originally by Nir Soffer)
Comment 7 RHV Bugzilla Automation and Verification Bot 2018-08-06 09:47:39 EDT
WARN: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{'rhevm-4.2.z': '?'}', ]

For more info please contact: rhv-devops@redhat.comINFO: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{'rhevm-4.2.z': '?'}', ]

For more info please contact: rhv-devops@redhat.com

(Originally by rhv-bugzilla-bot)
Comment 11 Kevin Alon Goldblatt 2018-08-09 05:41 EDT
Created attachment 1474619 [details]
engine and vdsm logs

Adding logs
Comment 13 Kevin Alon Goldblatt 2018-08-09 07:59:03 EDT
Based on Comment 12 and result from comment 8 moving to VERIFIED!
Comment 15 errata-xmlrpc 2018-09-04 09:43:24 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2624

Note You need to log in before you can comment on or make changes to this bug.