+++ This bug is a downstream clone. The original bug is: +++ +++ bug 1560460 +++ ====================================================================== Description of problem: NetApp snapshots feature may include symbolic links to NFS share which points to the same directory. This causes getFileStats to fail. Example: /rhev/data-center/mnt/<mountPoint>/.snapshot/daily.2018-02-20_0010 -> /rhev/data-center/mnt/<mountPoint>/<SD_UUID>/images/11111111-1111-1111-1111-111111111111/ Version-Release number of selected component (if applicable): python-ioprocess-0.16.1-1.el7ev.noarch ioprocess-0.16.1-1.el7ev.x86_64 How reproducible: 100% Steps to Reproduce: 1. Go to NFS ISO domain and create a recursive symbolic link. # pwd /rhev/data-center/mnt/10.34.1.195:_exports_rheviso01/c217bcfa-ffba-4982-b2c3-4ed418a0cd1d/images/11111111-1111-1111-1111-111111111111 # ln -s ../ testLink 2. Try to get the list of isos in the WebAdmin Actual results: 2018-03-26 10:08:03,596+0200 ERROR (jsonrpc/6) [storage.TaskManager.Task] (Task='70ba66b9-dd38-4042-9baa -d390a8c44305') Unexpected error (task:872) Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 879, in _run return fn(*args, **kargs) File "<string>", line 2, in getFileStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 2305, in getFileStats caseSensitive=caseSensitive) File "/usr/share/vdsm/storage/fileSD.py", line 496, in getFileList filesList = self.oop.simpleWalk(basedir) File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 351, in simpleWalk files.extend(simpleWalk(ioproc, fullpath)) File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 351, in simpleWalk files.extend(simpleWalk(ioproc, fullpath)) File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 351, in simpleWalk files.extend(simpleWalk(ioproc, fullpath)) .... File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 255, in isdir res = self._iop.stat(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 484, in stat resdict = self._sendCommand("stat", {"path": path}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 461, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 40] Too many levels of symbolic links .... 2018-03-26 10:08:03,600+0200 ERROR (jsonrpc/6) [storage.Dispatcher] FINISH getFileStats error=[Errno 40] Too many levels of symbolic links (dispatcher:85) Traceback (most recent call last): File "/usr/share/vdsm/storage/dispatcher.py", line 72, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/share/vdsm/storage/task.py", line 105, in wrapper return m(self, *a, **kw) File "/usr/share/vdsm/storage/task.py", line 1185, in prepare raise self.error OSError: [Errno 40] Too many levels of symbolic links Expected results: The symbolic link is skipped. Additional info: ioprocess tries to skip the symlinks: def simpleWalk(ioproc, path): files = [] for f in ioproc.listdir(path): fullpath = os.path.join(path, f) osPath = _IOProcessOs(ioproc).path if osPath.isdir(fullpath) and not osPath.islink(fullpath): files.extend(simpleWalk(ioproc, fullpath)) else: files.append(fullpath) return files The problem is that the islink function uses "stat" function from ioprocess c library which does use a stat from sys/stat.h which returns information about the target not about the link itself. So lstat function should be used instead. (Originally by Roman Hodain)
Roman, can you share output of "ls -lhR" in /rhev/data-center/mnt/server:_path for a faulty server using these recursive symlinks? (Originally by Nir Soffer)
We don't have enough data to understand how a recursive link was created inside a the iso domain images directory, but checking the source show that we indeed implement os.path.islink() in the wrong way in vdsm. (Originally by Nir Soffer)
Fixing this requires adding lstat() support in ioprocess, see bug 1564515. (Originally by Nir Soffer)
WARN: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason: [Found non-acked flags: '{'rhevm-4.2.z': '?'}', ] For more info please contact: rhv-devops: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason: [Found non-acked flags: '{'rhevm-4.2.z': '?'}', ] For more info please contact: rhv-devops (Originally by rhv-bugzilla-bot)
Created attachment 1474619 [details] engine and vdsm logs Adding logs
Based on Comment 12 and result from comment 8 moving to VERIFIED!
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:2624
sync2jira