Bug 965266
| Summary: | Create non-recursive not supported (HBASE) | |||
|---|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Jay Vyas <jvyas> | |
| Component: | gluster-hadoop | Assignee: | Jay Vyas <jvyas> | |
| Status: | CLOSED WONTFIX | QA Contact: | Martin Bukatovic <mbukatov> | |
| Severity: | medium | Docs Contact: | ||
| Priority: | medium | |||
| Version: | mainline | CC: | aavati, bchilds, dahorak, eboyd, matt, mbukatov, rhs-bugs, vbellur | |
| Target Milestone: | --- | Keywords: | Triaged | |
| Target Release: | --- | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | Bug Fix | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 968432 974241 (view as bug list) | Environment: | ||
| Last Closed: | 2014-04-28 23:37:01 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 968432, 969195, 969223, 974241, 1057253, 1084239 | |||
|
Description
Jay Vyas
2013-05-20 19:40:58 UTC
This has been patched and is awaiting review based on another HCFS implementation (quantcast) This is now implemented and working with HBASE https://github.com/gluster/hadoop-glusterfs/commit/fe1dfdfb77168e103c2732e7a652a376632e3a67 The above implementation only supports 1.x here is why:
In hadoop 1.x, there are 2 implementations of createNonRecursive:
[root@jvyas hadoop-common]# git checkout branch-1.1
Switched to branch 'branch-1.1'
[root@jvyas hadoop-common]# grep NonRec ./src/core/org/apache/hadoop/fs/FileSystem.java
public FSDataOutputStream createNonRecursive(Path f,
return this.createNonRecursive(f, FsPermission.getDefault(),
public FSDataOutputStream createNonRecursive(Path f, FsPermission permission,
throw new IOException("createNonRecursive unsupported for this filesystem "
But alas, when we go to hadoop 2.x, there are 3 !
[root@jvyas hadoop-common]# git checkout branch-2.1.0-alpha
Switched to branch 'branch-2.1.0-alpha'
[root@jvyas hadoop-common]# cat ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java | grep NonRec
public FSDataOutputStream createNonRecursive(Path f,
return this.createNonRecursive(f, FsPermission.getDefault(),
public FSDataOutputStream createNonRecursive(Path f, FsPermission permission,
return createNonRecursive(f, permission,
public FSDataOutputStream createNonRecursive(Path f, FsPermission permission,
throw new IOException("createNonRecursive unsupported for this filesystem "
###################################
So , in summary, we will also want to implement this signature so that hbase works on top of hadoop 2.x core libraries.
Fixed with redesign in this checkin: https://github.com/gluster/hadoop-glusterfs/commit/ce6325313dcb0df6cc73379248c1e07a9aa0b025 Are we sure that this is necessarily a fix ? Possible wrinkle: Since the underlying file system implementation will rely on the RawLocalFileSystem implementation, its possible that if we do not implement our own createNonRecursive, there will be variable implementation of the underlying method in the inherited class, which will result in variable implementation of createNonRecursive in the runtime glusterfs-hadoop plugin. Ive reopened and confirmed that this is still a bug, and also , reimplemented a pull request for the patch. It involves retroadding in the createNonRecursive implementation so that 1.x implementatinos of RawLocalFileSystem, which arent API compliant with hbase 94.7, still work https://github.com/gluster/glusterfs-hadoop/pull/60 https://github.com/jayunit100/glusterfs-hadoop/commit/6a40c962a3d0adb2a2c33c16ccb2b80a77489781 strongly suggest quick review +1 so we can restore hbase functionality against 1.x and 2.x again. STEPS TO REPRODUCE: 1) Install HBASE with the underlying hadoop library hadoop-core-1.0.4 on top of gluster as the distributed file system. 2) Create a table i.e. "create 't1','f1'". Side Note: I beleive that the "createNonRecursive" API call is only excersized in distributed mode, I beleive because RegionServers arent written to in local mode. Jay, The createNonRecursive(...) methods are deprecated for the FileSystem API. My theory is that HBASE later versions + hadoop 2.x may not even need the method anymore. I'm assigning to you for further investigation. If we must have the createNonRecursive method please provide a pull. Non issue for now: We don't support HBASE on 1x anymore. Non issue for now: We don't support HBASE on 1x anymore, so this isnt a problem or an issue we're concerned about. |