Upon HBASE (0.94.7) table creation, an interesting error surface which points to an unimplemented feature in the glusterfs plugin: "createNonRecursive" not supported. This method is deprecated in current hadoop : http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html but http://apache.cs.utah.edu/hbase/stable/ points to 0.94.7 as the "stable" hbase implementation. TO FIX: Implement the deprectated functionality in the glusterfs-hadoop plugin. -------------- LOGS ------------------ 2013-05-20 15:08:42,375 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Exception in region server : java.io.IOException: Exception in createWriter at org.apache.hadoop.hbase.regionserver.wal.HLogFileSystem.createWriter(HLogFileSystem.java:66) at org.apache.hadoop.hbase.regionserver.wal.HLog.createWriterInstance(HLog.java:715) at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:648) at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:592) at org.apache.hadoop.hbase.regionserver.wal.HLog.<init>(HLog.java:446) at org.apache.hadoop.hbase.regionserver.wal.HLog.<init>(HLog.java:370) at org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateHLog(HRegionServer.java:1410) at org.apache.hadoop.hbase.regionserver.HRegionServer.setupWALAndReplication(HRegionServer.java:1381) at org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1094) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:741) at java.lang.Thread.run(Thread.java:679) Caused by: java.io.IOException: cannot get log writer at org.apache.hadoop.hbase.regionserver.wal.HLog.createWriter(HLog.java:771) at org.apache.hadoop.hbase.regionserver.wal.HLogFileSystem.createWriter(HLogFileSystem.java:60) ... 10 more Caused by: java.io.IOException: java.io.IOException: createNonRecursive unsupported for this filesystem class org.apache.hadoop.fs.glusterfs.GlusterFileSystem at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.init(SequenceFileLogWriter.java:173) at org.apache.hadoop.hbase.regionserver.wal.HLog.createWriter(HLog.java:768) ... 11 more Caused by: java.io.IOException: createNonRecursive unsupported for this filesystem class org.apache.hadoop.fs.glusterfs.GlusterFileSystem at org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:626) at org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:601) at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:442) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.init(SequenceFileLogWriter.java:154) ... 12 more 2013-05-20 15:08:42,376 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Failed initialization 2013-05-20 15:08:42,377 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: Failed init java.io.IOException: Exception in createWriter at org.apache.hadoop.hbase.regionserver.wal.HLogFileSystem.createWriter(HLogFileSystem.java:66) at org.apache.hadoop.hbase.regionserver.wal.HLog.createWriterInstance(HLog.java:715) at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:648) at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:592) at org.apache.hadoop.hbase.regionserver.wal.HLog.<init>(HLog.java:446) at org.apache.hadoop.hbase.regionserver.wal.HLog.<init>(HLog.java:370) at org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateHLog(HRegionServer.java:1410) at org.apache.hadoop.hbase.regionserver.HRegionServer.setupWALAndReplication(HRegionServer.java:1381) at org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1094) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:741) at java.lang.Thread.run(Thread.java:679) Caused by: java.io.IOException: cannot get log writer at org.apache.hadoop.hbase.regionserver.wal.HLog.createWriter(HLog.java:771) at org.apache.hadoop.hbase.regionserver.wal.HLogFileSystem.createWriter(HLogFileSystem.java:60) ... 10 more Caused by: java.io.IOException: java.io.IOException: createNonRecursive unsupported for this filesystem class org.apache.hadoop.fs.glusterfs.GlusterFileSystem at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.init(SequenceFileLogWriter.java:173) at org.apache.hadoop.hbase.regionserver.wal.HLog.createWriter(HLog.java:768) ... 11 more Caused by: java.io.IOException: createNonRecursive unsupported for this filesystem class org.apache.hadoop.fs.glusterfs.GlusterFileSystem at org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:626) at org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:601) at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:442) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.init(SequenceFileLogWriter.java:154) ... 12 more 2013-05-20 15:08:42,379 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server hbase-master,60020,1369076864864: Unhandled exception: Exception in createWriter java.io.IOException: Exception in createWriter at org.apache.hadoop.hbase.regionserver.wal.HLogFileSystem.createWriter(HLogFileSystem.java:66) at org.apache.hadoop.hbase.regionserver.wal.HLog.createWriterInstance(HLog.java:715) at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:648) at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:592) at org.apache.hadoop.hbase.regionserver.wal.HLog.<init>(HLog.java:446) at org.apache.hadoop.hbase.regionserver.wal.HLog.<init>(HLog.java:370) at org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateHLog(HRegionServer.java:1410) at org.apache.hadoop.hbase.regionserver.HRegionServer.setupWALAndReplication(HRegionServer.java:1381) at org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1094) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:741) at java.lang.Thread.run(Thread.java:679) Caused by: java.io.IOException: cannot get log writer at org.apache.hadoop.hbase.regionserver.wal.HLog.createWriter(HLog.java:771) at org.apache.hadoop.hbase.regionserver.wal.HLogFileSystem.createWriter(HLogFileSystem.java:60) ... 10 more Caused by: java.io.IOException: java.io.IOException: createNonRecursive unsupported for this filesystem class org.apache.hadoop.fs.glusterfs.GlusterFileSystem at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.init(SequenceFileLogWriter.java:173) at org.apache.hadoop.hbase.regionserver.wal.HLog.createWriter(HLog.java:768) ... 11 more Caused by: java.io.IOException: createNonRecursive unsupported for this filesystem class org.apache.hadoop.fs.glusterfs.GlusterFileSystem at org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:626) at org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:601) at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:442) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.init(SequenceFileLogWriter.java:154) ... 12 more
This has been patched and is awaiting review based on another HCFS implementation (quantcast)
This is now implemented and working with HBASE https://github.com/gluster/hadoop-glusterfs/commit/fe1dfdfb77168e103c2732e7a652a376632e3a67
The above implementation only supports 1.x here is why: In hadoop 1.x, there are 2 implementations of createNonRecursive: [root@jvyas hadoop-common]# git checkout branch-1.1 Switched to branch 'branch-1.1' [root@jvyas hadoop-common]# grep NonRec ./src/core/org/apache/hadoop/fs/FileSystem.java public FSDataOutputStream createNonRecursive(Path f, return this.createNonRecursive(f, FsPermission.getDefault(), public FSDataOutputStream createNonRecursive(Path f, FsPermission permission, throw new IOException("createNonRecursive unsupported for this filesystem " But alas, when we go to hadoop 2.x, there are 3 ! [root@jvyas hadoop-common]# git checkout branch-2.1.0-alpha Switched to branch 'branch-2.1.0-alpha' [root@jvyas hadoop-common]# cat ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java | grep NonRec public FSDataOutputStream createNonRecursive(Path f, return this.createNonRecursive(f, FsPermission.getDefault(), public FSDataOutputStream createNonRecursive(Path f, FsPermission permission, return createNonRecursive(f, permission, public FSDataOutputStream createNonRecursive(Path f, FsPermission permission, throw new IOException("createNonRecursive unsupported for this filesystem " ################################### So , in summary, we will also want to implement this signature so that hbase works on top of hadoop 2.x core libraries.
Fixed with redesign in this checkin: https://github.com/gluster/hadoop-glusterfs/commit/ce6325313dcb0df6cc73379248c1e07a9aa0b025
Are we sure that this is necessarily a fix ? Possible wrinkle: Since the underlying file system implementation will rely on the RawLocalFileSystem implementation, its possible that if we do not implement our own createNonRecursive, there will be variable implementation of the underlying method in the inherited class, which will result in variable implementation of createNonRecursive in the runtime glusterfs-hadoop plugin.
Ive reopened and confirmed that this is still a bug, and also , reimplemented a pull request for the patch. It involves retroadding in the createNonRecursive implementation so that 1.x implementatinos of RawLocalFileSystem, which arent API compliant with hbase 94.7, still work https://github.com/gluster/glusterfs-hadoop/pull/60 https://github.com/jayunit100/glusterfs-hadoop/commit/6a40c962a3d0adb2a2c33c16ccb2b80a77489781 strongly suggest quick review +1 so we can restore hbase functionality against 1.x and 2.x again.
STEPS TO REPRODUCE: 1) Install HBASE with the underlying hadoop library hadoop-core-1.0.4 on top of gluster as the distributed file system. 2) Create a table i.e. "create 't1','f1'". Side Note: I beleive that the "createNonRecursive" API call is only excersized in distributed mode, I beleive because RegionServers arent written to in local mode.
Jay, The createNonRecursive(...) methods are deprecated for the FileSystem API. My theory is that HBASE later versions + hadoop 2.x may not even need the method anymore. I'm assigning to you for further investigation. If we must have the createNonRecursive method please provide a pull.
Non issue for now: We don't support HBASE on 1x anymore.
Non issue for now: We don't support HBASE on 1x anymore, so this isnt a problem or an issue we're concerned about.