Description of problem: NetApp snapshots feature may include symbolic links to NFS share which points to the same directory. This causes getFileStats to fail. Example: /rhev/data-center/mnt/<mountPoint>/.snapshot/daily.2018-02-20_0010 -> /rhev/data-center/mnt/<mountPoint>/<SD_UUID>/images/11111111-1111-1111-1111-111111111111/ Version-Release number of selected component (if applicable): python-ioprocess-0.16.1-1.el7ev.noarch ioprocess-0.16.1-1.el7ev.x86_64 How reproducible: 100% Steps to Reproduce: 1. Go to NFS ISO domain and create a recursive symbolic link. # pwd /rhev/data-center/mnt/10.34.1.195:_exports_rheviso01/c217bcfa-ffba-4982-b2c3-4ed418a0cd1d/images/11111111-1111-1111-1111-111111111111 # ln -s ../ testLink 2. Try to get the list of isos in the WebAdmin Actual results: 2018-03-26 10:08:03,596+0200 ERROR (jsonrpc/6) [storage.TaskManager.Task] (Task='70ba66b9-dd38-4042-9baa -d390a8c44305') Unexpected error (task:872) Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 879, in _run return fn(*args, **kargs) File "<string>", line 2, in getFileStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 2305, in getFileStats caseSensitive=caseSensitive) File "/usr/share/vdsm/storage/fileSD.py", line 496, in getFileList filesList = self.oop.simpleWalk(basedir) File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 351, in simpleWalk files.extend(simpleWalk(ioproc, fullpath)) File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 351, in simpleWalk files.extend(simpleWalk(ioproc, fullpath)) File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 351, in simpleWalk files.extend(simpleWalk(ioproc, fullpath)) .... File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 255, in isdir res = self._iop.stat(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 484, in stat resdict = self._sendCommand("stat", {"path": path}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 461, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 40] Too many levels of symbolic links .... 2018-03-26 10:08:03,600+0200 ERROR (jsonrpc/6) [storage.Dispatcher] FINISH getFileStats error=[Errno 40] Too many levels of symbolic links (dispatcher:85) Traceback (most recent call last): File "/usr/share/vdsm/storage/dispatcher.py", line 72, in wrapper result = ctask.prepare(func, *args, **kwargs) File "/usr/share/vdsm/storage/task.py", line 105, in wrapper return m(self, *a, **kw) File "/usr/share/vdsm/storage/task.py", line 1185, in prepare raise self.error OSError: [Errno 40] Too many levels of symbolic links Expected results: The symbolic link is skipped. Additional info: ioprocess tries to skip the symlinks: def simpleWalk(ioproc, path): files = [] for f in ioproc.listdir(path): fullpath = os.path.join(path, f) osPath = _IOProcessOs(ioproc).path if osPath.isdir(fullpath) and not osPath.islink(fullpath): files.extend(simpleWalk(ioproc, fullpath)) else: files.append(fullpath) return files The problem is that the islink function uses "stat" function from ioprocess c library which does use a stat from sys/stat.h which returns information about the target not about the link itself. So lstat function should be used instead.
Roman, can you share output of "ls -lhR" in /rhev/data-center/mnt/server:_path for a faulty server using these recursive symlinks?
We don't have enough data to understand how a recursive link was created inside a the iso domain images directory, but checking the source show that we indeed implement os.path.islink() in the wrong way in vdsm.
Fixing this requires adding lstat() support in ioprocess, see bug 1564515.
WARN: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason: [Found non-acked flags: '{'rhevm-4.2.z': '?'}', ] For more info please contact: rhv-devops: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason: [Found non-acked flags: '{'rhevm-4.2.z': '?'}', ] For more info please contact: rhv-devops
Verify according to https://bugzilla.redhat.com/show_bug.cgi?id=1612904#c13
Verified with the following code: ---------------------------------------- ovirt-engine-4.3.0-0.0.master.20181012165724.gitd25f971.el7.noarch vdsm-4.30.0-640.git6fd8327.el7.x86_64 Verified with the following scenario: ---------------------------------------- Steps to Reproduce: 1. Go to NFS ISO domain and create a recursive symbolic link. # pwd /rhev/data-center/mnt/10.34.1.195:_exports_rheviso01/c217bcfa-ffba-4982-b2c3-4ed418a0cd1d/images/11111111-1111-1111-1111-111111111111 # ln -s ../ testLink 2. Try to get the list of isos in the WebAdmin >>>>> Works fine and no errors are reported
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1077
sync2jira