Created attachment 990809 [details] NFS client log Description of problem: ======================= Mounted a 1+(4*2) disperse volume on client and did a 'ls -l'. Seeing Input/Output errors even though its listing the directories. When tried to cd to one of the directory and doesn't let in and displays I/O error. [root@dhcp37-205 fuse1]# mount -t nfs vertigo:/testvol /mnt/nfs [root@dhcp37-205 fuse1]# cd /mn^C [root@dhcp37-205 fuse1]# cd /mnt/nfs [root@dhcp37-205 nfs]# ls -l ls: linux: Input/output error ls: files: Input/output error ls: nfs_files: Input/output error ls: newfiles: Input/output error ls: newdirs: Input/output error total 8152 drwxr-xr-x. 332521 root root 8011776 Feb 7 12:08 dirs drwxr-xr-x. 2 root root 253952 Feb 5 16:56 files drwxr-xr-x. 13 root root 8192 Feb 6 16:26 linux drwxr-xr-x. 3 root root 18 Feb 12 14:24 newdirs drwxr-xr-x. 1033 root root 32768 Feb 5 18:04 newfiles drwxr-xr-x. 1416 root root 40960 Feb 11 20:52 nfs_files [root@dhcp37-205 nfs]# [root@dhcp37-205 nfs]# cd dirs/ [root@dhcp37-205 dirs]# cd ../files bash: cd: ../files: Input/output error [root@dhcp37-205 dirs]# cd ../linux bash: cd: ../linux: Input/output error [root@dhcp37-205 dirs]# cd ../newdirs bash: cd: ../newdirs: Input/output error [root@dhcp37-205 dirs]# A bug 1187526 was filed and merged on 5th feb for the NFS issue. Version-Release number of selected component (if applicable): ============================================================= glusterfs 3.7dev built on Feb 8 2015 01:04:29 Gluster volume options: ======================= [root@vertigo ~]# gluster v get testvol all Option Value ------ ----- cluster.lookup-unhashed on cluster.min-free-disk 10% cluster.min-free-inodes 5% cluster.rebalance-stats off cluster.subvols-per-directory (null) cluster.readdir-optimize off cluster.rsync-hash-regex (null) cluster.extra-hash-regex (null) cluster.dht-xattr-name trusted.glusterfs.dht cluster.randomize-hash-range-by-gfid off cluster.local-volume-name (null) cluster.weighted-rebalance on cluster.switch-pattern (null) cluster.entry-change-log on cluster.read-subvolume (null) cluster.read-subvolume-index -1 cluster.read-hash-mode 1 cluster.background-self-heal-count 16 cluster.metadata-self-heal on cluster.data-self-heal on cluster.entry-self-heal on cluster.self-heal-daemon on cluster.heal-timeout 600 cluster.self-heal-window-size 1 cluster.data-change-log on cluster.metadata-change-log on cluster.data-self-heal-algorithm (null) cluster.eager-lock on cluster.quorum-type none cluster.quorum-count (null) cluster.choose-local true cluster.self-heal-readdir-size 1KB cluster.post-op-delay-secs 1 cluster.ensure-durability on cluster.stripe-block-size 128KB cluster.stripe-coalesce true diagnostics.latency-measurement off diagnostics.dump-fd-stats off diagnostics.count-fop-hits off diagnostics.brick-log-level INFO diagnostics.client-log-level INFO diagnostics.brick-sys-log-level CRITICAL diagnostics.client-sys-log-level CRITICAL diagnostics.brick-logger (null) diagnostics.client-logger (null) diagnostics.brick-log-format (null) diagnostics.client-log-format (null) diagnostics.brick-log-buf-size 5 diagnostics.client-log-buf-size 5 diagnostics.brick-log-flush-timeout 120 diagnostics.client-log-flush-timeout 120 performance.cache-max-file-size 0 performance.cache-min-file-size 0 performance.cache-refresh-timeout 1 performance.cache-priority performance.cache-size 32MB performance.io-thread-count 16 performance.high-prio-threads 16 performance.normal-prio-threads 16 performance.low-prio-threads 16 performance.least-prio-threads 1 performance.enable-least-priority on performance.least-rate-limit 0 performance.cache-size 128MB performance.flush-behind on performance.nfs.flush-behind on performance.write-behind-window-size 1MB performance.nfs.write-behind-window-size1MB performance.strict-o-direct off performance.nfs.strict-o-direct off performance.strict-write-ordering off performance.nfs.strict-write-ordering off performance.lazy-open yes performance.read-after-open no performance.read-ahead-page-count 4 performance.md-cache-timeout 1 features.encryption off encryption.master-key (null) encryption.data-key-size 256 encryption.block-size 4096 network.frame-timeout 1800 network.ping-timeout 42 network.tcp-window-size (null) features.lock-heal off features.grace-timeout 10 network.remote-dio disable client.event-threads 2 network.tcp-window-size (null) network.inode-lru-limit 16384 auth.allow * auth.reject (null) transport.keepalive (null) server.allow-insecure (null) server.root-squash off server.anonuid 65534 server.anongid 65534 server.statedump-path /var/run/gluster server.outstanding-rpc-limit 64 features.lock-heal off features.grace-timeout (null) server.ssl (null) auth.ssl-allow * server.manage-gids off client.send-gids on server.gid-timeout 2 server.own-thread (null) server.event-threads 2 performance.write-behind on performance.read-ahead on performance.readdir-ahead off performance.io-cache on performance.quick-read on performance.open-behind on performance.stat-prefetch on performance.client-io-threads off performance.nfs.write-behind on performance.nfs.read-ahead off performance.nfs.io-cache off performance.nfs.quick-read off performance.nfs.stat-prefetch off performance.nfs.io-threads off performance.force-readdirp true features.file-snapshot off features.uss off features.snapshot-directory .snaps features.show-snapshot-directory off network.compression off network.compression.window-size -15 network.compression.mem-level 8 network.compression.min-size 0 network.compression.compression-level -1 network.compression.debug false features.limit-usage (null) features.quota-timeout 0 features.default-soft-limit 80% features.soft-timeout 60 features.hard-timeout 5 features.alert-time 86400 features.quota-deem-statfs off geo-replication.indexing off geo-replication.indexing off geo-replication.ignore-pid-check off geo-replication.ignore-pid-check off features.quota on debug.trace off debug.log-history no debug.log-file no debug.exclude-ops (null) debug.include-ops (null) debug.error-gen off debug.error-failure (null) debug.error-number (null) debug.random-failure off debug.error-fops (null) nfs.enable-ino32 no nfs.mem-factor 15 nfs.export-dirs on nfs.export-volumes on nfs.addr-namelookup off nfs.dynamic-volumes off nfs.register-with-portmap on nfs.outstanding-rpc-limit 16 nfs.port 2049 nfs.rpc-auth-unix on nfs.rpc-auth-null on nfs.rpc-auth-allow all nfs.rpc-auth-reject none nfs.ports-insecure off nfs.trusted-sync off nfs.trusted-write off nfs.volume-access read-write nfs.export-dir nfs.disable false nfs.nlm on nfs.acl on nfs.mount-udp off nfs.mount-rmtab /var/lib/glusterd/nfs/rmtab nfs.rpc-statd /sbin/rpc.statd nfs.server-aux-gids off nfs.drc off nfs.drc-size 0x20000 nfs.read-size (1 * 1048576ULL) nfs.write-size (1 * 1048576ULL) nfs.readdir-size (1 * 1048576ULL) features.read-only off features.worm off storage.linux-aio off storage.batch-fsync-mode reverse-fsync storage.batch-fsync-delay-usec 0 storage.owner-uid -1 storage.owner-gid -1 storage.node-uuid-pathinfo off storage.health-check-interval 30 storage.build-pgfid off storage.bd-aio off cluster.server-quorum-type off cluster.server-quorum-ratio 0 changelog.changelog off changelog.changelog-dir (null) changelog.encoding ascii changelog.rollover-time 15 changelog.fsync-interval 5 changelog.changelog-barrier-timeout 120 features.barrier disable features.barrier-timeout 120 locks.trace disable cluster.disperse-self-heal-daemon enable [root@vertigo ~]# Gluster volume info: ==================== [root@vertigo ~]# gluster v info Volume Name: testvol Type: Disperse Volume ID: 21ed8908-3458-4834-b93d-161b694c3e37 Status: Started Number of Bricks: 1 x (4 + 2) = 6 Transport-type: tcp Bricks: Brick1: ninja:/rhs/brick1/b1 Brick2: vertigo:/rhs/brick1/b1 Brick3: ninja:/rhs/brick2/b2 Brick4: vertigo:/rhs/brick2/b2 Brick5: ninja:/rhs/brick3/b3 Brick6: vertigo:/rhs/brick3/b3 Options Reconfigured: cluster.disperse-self-heal-daemon: enable features.quota: on [root@vertigo ~]# Gluster volume status: ====================== [root@vertigo ~]# gluster v status Status of volume: testvol Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick ninja:/rhs/brick1/b1 49152 Y 13029 Brick vertigo:/rhs/brick1/b1 49152 Y 13852 Brick ninja:/rhs/brick2/b2 49153 Y 13041 Brick vertigo:/rhs/brick2/b2 49153 Y 13864 Brick ninja:/rhs/brick3/b3 49154 Y 13053 Brick vertigo:/rhs/brick3/b3 49158 Y 13876 NFS Server on localhost 2049 Y 13891 Quota Daemon on localhost N/A Y 13907 NFS Server on 10.70.34.68 2049 Y 13068 Quota Daemon on 10.70.34.68 N/A Y 13085 Task Status of Volume testvol ------------------------------------------------------------------------------ There are no active volume tasks [root@vertigo ~]# How reproducible: ================= 100% Steps to Reproduce: 1. Mount a gluster volume using NFS 2. Try to list the files/directories with 'ls -l' 3. Try to cd to any of the directory Actual results: =============== Input/Output error Expected results: ================= No errors should be seen Additional info: Attaching the client nfs log snippet when the actions are performed.
Bhaskar, I tried doing the following, but not able to recreate the issue. Could you let me know how the directories on the mount are created? root@pranithk-laptop - /mnt 12:37:58 :( ⚡ glusterd && gluster volume create ec2 disperse 6 redundancy 2 `hostname`:/home/gfs/ec_{0,1,2,3,4,5} force && gluster volume start ec2 volume create: ec2: success: please start the volume to access data volume start: ec2: success root@pranithk-laptop - /mnt 12:38:15 :) ⚡ mount -t glusterfs `hostname`:/ec2 /mnt/ec2 root@pranithk-laptop - /mnt 12:38:23 :) ⚡ cd /mnt/ec2 root@pranithk-laptop - /mnt/ec2 12:38:24 :) ⚡ mkdir 1 2 3 4 5 root@pranithk-laptop - /mnt/ec2 12:38:27 :) ⚡ touch a v c d e f root@pranithk-laptop - /mnt/ec2 12:38:29 :) ⚡ cd root@pranithk-laptop - ~ 12:38:33 :) ⚡ mount -t nfs -o vers=3 `hostname`:/ec2 /mnt/nfs root@pranithk-laptop - ~ 12:38:40 :) ⚡ cd /mnt/nfs root@pranithk-laptop - /mnt/nfs 12:38:41 :) ⚡ root@pranithk-laptop - /mnt/nfs 12:38:41 :) ⚡ ls -l total 3 drwxr-xr-x. 2 root root 6 Mar 9 12:38 1 drwxr-xr-x. 2 root root 6 Mar 9 12:38 2 drwxr-xr-x. 2 root root 6 Mar 9 12:38 3 drwxr-xr-x. 2 root root 6 Mar 9 12:38 4 drwxr-xr-x. 2 root root 6 Mar 9 12:38 5 -rw-r--r--. 1 root root 0 Mar 9 12:38 a -rw-r--r--. 1 root root 0 Mar 9 12:38 c -rw-r--r--. 1 root root 0 Mar 9 12:38 d -rw-r--r--. 1 root root 0 Mar 9 12:38 e -rw-r--r--. 1 root root 0 Mar 9 12:38 f -rw-r--r--. 1 root root 0 Mar 9 12:38 v root@pranithk-laptop - /mnt/nfs 12:38:43 :) ⚡ Pranith
Just saw the nfs logs. The errors were happening because of nameless loc. [2015-02-12 09:00:01.267287] E [ec-helpers.c:391:ec_loc_setup_path] 0-testvol-disperse-0: Invalid path '<gfid:41735a87-d89f-4c81-91d9-c59efc640f4d>' in loc This bug is fixed as part of http://review.gluster.com/9717
Verified this on the latest mater and the issue is not seen.
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report. glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939 [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user