Bug 1447266

Summary: [snapshot cifs]ls on .snaps directory is throwing input/output error over cifs mount
Product: [Community] GlusterFS Reporter: Mohammed Rafi KC <rkavunga>
Component: snapshotAssignee: Mohammed Rafi KC <rkavunga>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: mainlineCC: amukherj, anoopcs, bugs, rhs-bugs, rhs-smb, rkavunga, sbhaloth, storage-qa-internal, vdas
Target Milestone: ---Keywords: Regression, Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.12.0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1441280
: 1463522 1530341 (view as bug list) Environment:
Last Closed: 2017-09-05 17:28:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1441280, 1463522, 1530341    

Comment 2 Mohammed Rafi KC 2017-05-02 09:33:21 UTC
Description of problem:
ls on .snaps directory is throwing input/output error over cifs mount.
Listing works on fuse mount and windows mount.

Version-Release number of selected component (if applicable):
mainline

How reproducible:
Always

Steps to Reproduce:
1.Enable uss & VSS plugins
2.Mount cifs on a samba ctdb gluster cluster
3.cd to the mount point
4. Do ll or ls over the cifs mount

Actual results:
ls: reading directory .snaps/: Input/output error

Expected results:
Should list out contents

Volume Name: saturday-saturday
Type: Distributed-Replicate
Volume ID: 4a24c34c-1144-4f07-9763-6e232c037a67
Status: Started
Snapshot Count: 0
Number of Bricks: 6 x 2 = 12
Transport-type: tcp
Options Reconfigured:
features.show-snapshot-directory: enable
features.uss: enable
transport.address-family: inet
nfs.disable: on
server.allow-insecure: on
performance.stat-prefetch: on
storage.batch-fsync-delay-usec: 0
features.cache-invalidation: on
features.cache-invalidation-timeout: 600
performance.cache-invalidation: on
performance.md-cache-timeout: 600
network.inode-lru-limit: 50000
performance.cache-samba-metadata: on
performance.parallel-readdir: on

smb.conf
---------

[gluster-saturday-saturday]
comment = For samba share of volume saturday-saturday
#vfs objects = glusterfs
vfs objects = shadow_copy2 glusterfs
glusterfs:volume = saturday-saturday
glusterfs:logfile = /var/log/samba/glusterfs-saturday-saturday.%M.log
glusterfs:loglevel = 9
shadow:snapdir = /.snaps
shadow:basedir = /
shadow:sort = desc
shadow:sscanf = false
#shadow:delimeter = _123john_cena_GMT
#shadow:snapprefix = [abc]
shadow:format = snap1_GMT-%Y.%m.%d-%H.%M.%S
path = /
read only = no
guest ok = yes

--- Additional comment from Anoop C S on 2017-04-18 07:45:40 EDT ---

I was able reproduce the bug easily. Following entries were seen in Samba logs:

[2017/04/12 15:42:25.377261,  5, pid=11231, effective(0, 0), real(0, 0)] ../source3/smbd/filename.c:644(unix_convert)
  unix_convert begin: name = .snaps/*, dirpath = .snaps, start = *
[2017/04/12 15:42:25.378556,  5, pid=11231, effective(0, 0), real(0, 0)] ../source3/smbd/filename.c:218(check_parent_exists)
  check_parent_exists: name = .snaps/*, dirpath = .snaps, start = *
[2017/04/12 15:42:25.378605, 10, pid=11231, effective(0, 0), real(0, 0)] ../source3/smbd/mangle_hash2.c:418(is_mangled)
  is_mangled * ?
[2017/04/12 15:42:25.378630, 10, pid=11231, effective(0, 0), real(0, 0)] ../source3/smbd/mangle_hash2.c:357(is_mangled_component)
  is_mangled_component * (len 1) ?
[2017/04/12 15:42:25.378653,  5, pid=11231, effective(0, 0), real(0, 0)] ../source3/smbd/filename.c:847(unix_convert)
  Wildcard *
[2017/04/12 15:42:25.378676, 10, pid=11231, effective(0, 0), real(0, 0), class=vfs] ../source3/smbd/vfs.c:1199(check_reduced_name)
  check_reduced_name: check_reduced_name [.snaps/*] [/]
[2017/04/12 15:42:25.380977, 10, pid=11231, effective(0, 0), real(0, 0), class=vfs] ../source3/smbd/vfs.c:1259(check_reduced_name)
  check_reduced_name realpath [.snaps/*] -> [/.snaps/*]
[2017/04/12 15:42:25.380999,  5, pid=11231, effective(0, 0), real(0, 0), class=vfs] ../source3/smbd/vfs.c:1370(check_reduced_name)
  check_reduced_name: .snaps/* reduced to /.snaps/*
[2017/04/12 15:42:25.381010,  5, pid=11231, effective(0, 0), real(0, 0)] ../source3/smbd/trans2.c:2741(call_trans2findfirst)
  dir=.snaps, mask = *
[2017/04/12 15:42:25.381026,  5, pid=11231, effective(0, 0), real(0, 0)] ../source3/smbd/dir.c:474(dptr_create)
  dptr_create dir=.snaps
[2017/04/12 15:42:25.381499, 10, pid=11231, effective(0, 0), real(0, 0)] ../source3/smbd/open.c:100(smbd_check_access_rights)
  smbd_check_access_rights: root override on .snaps. Granting 0x1
[2017/04/12 15:42:25.383028,  4, pid=11231, effective(0, 0), real(0, 0), class=vfs] ../source3/smbd/vfs.c:874(vfs_ChDir)
  vfs_ChDir to .snaps
[2017/04/12 15:42:25.383966,  1, pid=11231, effective(0, 0), real(0, 0), class=vfs] ../source3/smbd/vfs.c:921(vfs_GetWd)
  vfs_GetWd: couldn't stat "." error No such file or directory (NFS problem ?)
[2017/04/12 15:42:25.383985,  4, pid=11231, effective(0, 0), real(0, 0), class=vfs] ../source3/smbd/vfs.c:885(vfs_ChDir)
  vfs_ChDir got /.snaps
[2017/04/12 15:42:25.383998, 10, pid=11231, effective(0, 0), real(0, 0), class=vfs] ../source3/smbd/vfs.c:1199(check_reduced_name)
  check_reduced_name: check_reduced_name [.] [/]
[2017/04/12 15:42:25.385272,  3, pid=11231, effective(0, 0), real(0, 0), class=vfs] ../source3/smbd/vfs.c:1239(check_reduced_name)
  check_reduce_name: couldn't get realpath for . (NT_STATUS_OBJECT_PATH_NOT_FOUND)
[2017/04/12 15:42:25.385286,  5, pid=11231, effective(0, 0), real(0, 0)] ../source3/smbd/filename.c:1248(check_name)
  check_name: name . failed with NT_STATUS_OBJECT_PATH_NOT_FOUND

Judging from the above log it seems that stat and realpath vfs calls to glusterfs after changing directory to .snaps failed. Corresponding glusterfs client log entries:

[2017-04-12 10:12:25.379506] D [MSGID: 0] [client-rpc-fops.c:2936:client3_3_lookup_cbk] 0-stack-trace: stack-address: 0x55b8761a29b0, xcube-snapd-client returned -1 error: No such file or directory [No such file or directory]
[2017-04-12 10:12:25.379535] D [snapview-client.c:289:gf_svc_lookup_cbk] 0-xcube-snapview-client: Lookup failed on snapview graph with error No such file or directory
[2017-04-12 10:12:25.379554] D [MSGID: 0] [snapview-client.c:329:gf_svc_lookup_cbk] 0-stack-trace: stack-address: 0x55b8761a29b0, xcube-snapview-client returned -1 error: No such file or directory [No such file or directory]
[2017-04-12 10:12:25.379581] D [MSGID: 0] [io-stats.c:2213:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x55b8761a29b0, xcube returned -1 error: No such file or directory [No such file or directory]

I have attached a simple gfapi reproducer. We need to see why those calls are failing on .snaps directory.

--- Additional comment from Anoop C S on 2017-04-18 07:46 EDT ---



--- Additional comment from Mohammed Rafi KC on 2017-04-20 08:01:18 EDT ---

RCA:

Currently, snapview server is not handling the dentry name "." and ".." . So it fails for dentries pointing to entry_point ie, .snaps or the snapshot names.

Comment 0 is private: false

Comment 3 Worker Ant 2017-05-04 12:11:23 UTC
REVIEW: https://review.gluster.org/17177 (gfapi: fix handling of dot and double dot in path) posted (#1) for review on master by mohammed rafi  kc (rkavunga)

Comment 4 Worker Ant 2017-05-05 10:22:36 UTC
REVIEW: https://review.gluster.org/17177 (gfapi: fix handling of dot and double dot in path) posted (#2) for review on master by mohammed rafi  kc (rkavunga)

Comment 5 Worker Ant 2017-05-09 09:25:01 UTC
REVIEW: https://review.gluster.org/17177 (gfapi: fix handling of dot and double dot in path) posted (#3) for review on master by mohammed rafi  kc (rkavunga)

Comment 6 Worker Ant 2017-05-09 11:00:03 UTC
REVIEW: https://review.gluster.org/17216 (tests:Addidng testcase for bug 1447266) posted (#1) for review on master by mohammed rafi  kc (rkavunga)

Comment 7 Worker Ant 2017-05-09 11:12:55 UTC
REVIEW: https://review.gluster.org/17216 (tests:Addidng testcase for bug 1447266) posted (#2) for review on master by mohammed rafi  kc (rkavunga)

Comment 8 Worker Ant 2017-05-09 13:03:54 UTC
REVIEW: https://review.gluster.org/17216 (tests:Addidng testcase for bug 1447266) posted (#3) for review on master by Niels de Vos (ndevos)

Comment 9 Worker Ant 2017-05-10 12:33:04 UTC
REVIEW: https://review.gluster.org/17237 (gfapi: fix handling of dot and double dot in path) posted (#1) for review on release-3.11 by mohammed rafi  kc (rkavunga)

Comment 10 Worker Ant 2017-05-10 12:33:09 UTC
REVIEW: https://review.gluster.org/17238 (tests/gfapi:Addidng testcase to check handling of "." and "..") posted (#1) for review on release-3.11 by mohammed rafi  kc (rkavunga)

Comment 11 Worker Ant 2017-05-10 12:38:32 UTC
REVIEW: https://review.gluster.org/17216 (tests/gfapi:Addidng testcase to check handling of "." and "..") posted (#4) for review on master by mohammed rafi  kc (rkavunga)

Comment 12 Worker Ant 2017-05-10 12:38:37 UTC
REVIEW: https://review.gluster.org/17177 (gfapi: fix handling of dot and double dot in path) posted (#4) for review on master by mohammed rafi  kc (rkavunga)

Comment 13 Worker Ant 2017-05-12 09:32:58 UTC
REVIEW: https://review.gluster.org/17216 (tests/gfapi:Adding testcase to check handling of "." and "..") posted (#5) for review on master by mohammed rafi  kc (rkavunga)

Comment 14 Worker Ant 2017-05-12 09:33:04 UTC
REVIEW: https://review.gluster.org/17177 (gfapi: fix handling of dot and double dot in path) posted (#5) for review on master by mohammed rafi  kc (rkavunga)

Comment 15 Worker Ant 2017-05-12 15:54:46 UTC
COMMIT: https://review.gluster.org/17216 committed in master by Jeff Darcy (jeff.us) 
------
commit 642f3e290ade6a5a7087816a5cf633083ef9f608
Author: Mohammed Rafi KC <rkavunga>
Date:   Tue May 9 16:11:52 2017 +0530

    tests/gfapi:Adding testcase to check handling of "." and ".."
    
    Adding a testcase to check the proper handling of "." and ".."
    in gfapi path.
    The patch which fix the issue is https://review.gluster.org/#/c/17177
    
    Change-Id: I5c9cceade30f7d8a3b451b5f34f1cf9815729c4a
    BUG: 1447266
    Signed-off-by: Mohammed Rafi KC <rkavunga>
    Reviewed-on: https://review.gluster.org/17216
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Reviewed-by: Niels de Vos <ndevos>
    Reviewed-by: Jeff Darcy <jeff.us>
    CentOS-regression: Gluster Build System <jenkins.org>

Comment 16 Worker Ant 2017-05-14 13:14:08 UTC
COMMIT: https://review.gluster.org/17177 committed in master by Jeff Darcy (jeff.us) 
------
commit 269e2ccf45ddc662d8373eb887ae6cef96e2ef37
Author: Mohammed Rafi KC <rkavunga>
Date:   Wed May 3 17:19:57 2017 +0530

    gfapi: fix handling of dot and double dot in path
    
    This patch is to handle "." and ".." in file path. Which means
    this special dentry names will be resolved before sending fops
    on the path.
    
    Change-Id: I5e92f6d1ad1412bf432eb2488e53fb7731edb013
    BUG: 1447266
    Signed-off-by: Mohammed Rafi KC <rkavunga>
    Reviewed-on: https://review.gluster.org/17177
    Smoke: Gluster Build System <jenkins.org>
    Reviewed-by: Niels de Vos <ndevos>
    CentOS-regression: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Reviewed-by: Jeff Darcy <jeff.us>

Comment 17 Shyamsundar 2017-09-05 17:28:29 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.12.0, please open a new bug report.

glusterfs-3.12.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/announce/2017-September/000082.html
[2] https://www.gluster.org/pipermail/gluster-users/

Comment 18 Worker Ant 2017-11-17 00:27:43 UTC
REVIEW: https://review.gluster.org/18792 (gfapi: fix handling of dot and double dot in path) posted (#1) for review on release-3.10 by Taehwa Lee

Comment 19 Worker Ant 2018-01-02 16:30:06 UTC
REVISION POSTED: https://review.gluster.org/18792 (gfapi: fix handling of dot and double dot in path) posted (#2) for review on release-3.10 by Shyamsundar Ranganathan