Description of problem: The QuickSlave IO behavior needs to be removed in the GlusterFileSystem class given the issues with directly access bricks. One onerous example would be that the files on the brick might could be inconsistent with the volume. The logic needs to be removed from the GlusterFileSystem class in methods such as initialize() and open().
*** Bug 957274 has been marked as a duplicate of this bug. ***
Okay this is removed in this pull https://github.com/gluster/hadoop-glusterfs/pull/30
Could you write here any example how to test this fix? Do you know how to invoke inconsistency on volume?
A suggestion - you could add the QuickSlave IO properties to the core-site (via ambari) and then you could run a job and verify (perhaps via strace or something similar) that the NodeManagers DOES NOT try to access data from the brick mount directly (instead of going through the FUSE mount)
Yes - you'd have to do some runtime tests to verify the absence of this feature.
Related documentation --------------------- See documentation for RHS 2.0[1] (when the feature was in tech preview) quick.slave.io is a gluster specific property from core-site.xml (off by default) Performance tunable option. If this option is set to On, the plugin will try to perform I/O directly from the disk file system (like ext3 or ext4) the file resides on. As a result, read performance improves and jobs run faster. [1] https://access.redhat.com/documentation/en-US/Red_Hat_Storage/2.0/html/Administration_Guide/ch19s04.html
Retrying on Red Hat Storage Server 3.0 iso with: glusterfs-3.6.0.22-1.el6rhs.x86_64 Hortonworks Hadoop distribution: Ambari: ambari-server-1.4.4.23-1.noarch Stack : HDP-2.0.6.GlusterFS hadoop-2.2.0.2.0.6.0-101.el6.x86_64 RHS Hadoop plugin: rhs-hadoop-2.3.2-6.noarch rhs-hadoop-install-1_25-1.el6rhs.noarch Verification ============ Setting 'quick.slave.io' to 'On' via Ambari REST api, value double checked via groovy script using Hadoop api. Then running terragen, terrasort, terravalidate (as bigtop user generating 1GB): ~~~ /usr/lib/hadoop/bin/hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.2.0.2.0.6.0-101.jar teragen 10000000 /tmp/terra.in /usr/lib/hadoop/bin/hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.2.0.2.0.6.0-101.jar terasort /tmp/terra.in /tmp/terra.out /usr/lib/hadoop/bin/hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.2.0.2.0.6.0-101.jar teravalidate /tmp/terra.out /tmp/terra.report ~~~ While executing following strace for every YarnChild process on the worker nodes: ~~~ strace -p $PID -f -e trace=open > yarnchild.$PID.strace 2>&1 & ~~~ And the shim doesn't access the files on brick directly: ~~~ $ grep '/mnt/brick1/HadoopVol1' *.strace | wc -l 0 $ grep '/mnt/glusterfs/HadoopVol1' *.strace | wc -l 69 ~~~ So it works as expected.
This solution is no longer available from Red Hat.