Description of problem: missing apache mapreduce/system tests. These can be taken directly from the hadoop source tree and integrated into the gluster build.
Paraphrase from steve's email 2/20/2013 : More specifically, we can - Look in hadoop 1.0.4 under src/test/org/apache/hadoop/fs/kfs and - look how they created an Impl and a Single JUnit Test Class - test various I/O operations for that impl. Seems to me we'd get some pretty good and quick test coverage by just converting that class for use with the Hadoop FS generated by our Gluster Plugin.
Transferred to brad so that he can implement the base junit tests. if need be jay can then augment w/ kfs tests
Good news: we were able to hack the staging build server to pass unit tests: http://ec2-54-243-59-213.compute-1.amazonaws.com:8080/job/glusterfs-hadoop/41/console Bad news: we had to hack it. So the details follow - here is the way I got it working: 1) [on the server] manually mounting the gluster volume on the server 2) [on the server]recursive chmod on (0777) the /mnt/glusterfs directory 3) [on the shim] disabling a privileges exception for the gluster fs "mount -t" command, which is done by the shim. Why did we have to do these hacks? Because as of now, we are running gluster as root. However, on the build server, this is not possible. So the deeper root cause of this is that we cannot currently run the Shim as non-root. Will need to ping with brad/steve/a2/ab/mattf to see what next steps are. Current status: rolled back the hacks (#3 above) so that we can solve this the right way. Estimating another day or two. However, this bug is now closed, because we have confirmed that - given an environment with proper privileges to mount files, these DFS tests pass: ****************************************************************************** Started by user jay vyas [EnvInject] - Loading node environment variables. [EnvInject] - Preparing an environment for the build. [EnvInject] - Keeping Jenkins system variables. [EnvInject] - Keeping Jenkins build variables. [EnvInject] - Executing and processing the following script content: export GLUSTER_VOLUME=HadoopVol export GLUSTER_HOST=ec2-54-243-59-213.compute-1.amazonaws.com echo "done setting environmental vars for gluster:" env > /tmp/debugenv.txt [jenkins] $ /bin/sh -xe /tmp/hudson2555655623032837510.sh + export GLUSTER_VOLUME=HadoopVol + GLUSTER_VOLUME=HadoopVol + export GLUSTER_HOST=ec2-54-243-59-213.compute-1.amazonaws.com + GLUSTER_HOST=ec2-54-243-59-213.compute-1.amazonaws.com + echo 'done setting environmental vars for gluster:' done setting environmental vars for gluster: + env [EnvInject] - Script executed successfully. [EnvInject] - Injecting contributions. Building in workspace /var/lib/jenkins/jobs/glusterfs-hadoop/workspace Checkout:workspace / /var/lib/jenkins/jobs/glusterfs-hadoop/workspace - hudson.remoting.LocalChannel@48b9e55c Using strategy: Default Last Built Revision: Revision 7fd94b1a72b073d85aeac91be2c4e003f350a71f (origin/master, origin/HEAD) Fetching changes from 1 remote Git repository Fetching upstream changes from origin Seen branch in repository origin/BZ908526 Seen branch in repository origin/BZ908898 Seen branch in repository origin/BZ911420 Seen branch in repository origin/HEAD Seen branch in repository origin/junit Seen branch in repository origin/junit_squashed Seen branch in repository origin/master Commencing build of Revision 7fd94b1a72b073d85aeac91be2c4e003f350a71f (origin/master, origin/HEAD) Checking out Revision 7fd94b1a72b073d85aeac91be2c4e003f350a71f (origin/master, origin/HEAD) Warning : There are multiple branch changesets here [workspace] $ /bin/sh -xe /tmp/hudson3228325285554877313.sh Parsing POMs [workspace] $ java -cp /var/lib/jenkins/plugins/maven-plugin/WEB-INF/lib/maven3-agent-1.2.jar:/var/lib/jenkins/tools/hudson.tasks.Maven_MavenInstallation/mavenjenkins/boot/plexus-classworlds-2.4.jar org.jvnet.hudson.maven3.agent.Maven3Main /var/lib/jenkins/tools/hudson.tasks.Maven_MavenInstallation/mavenjenkins /var/cache/jenkins/war/WEB-INF/lib/remoting-2.22.jar /var/lib/jenkins/plugins/maven-plugin/WEB-INF/lib/maven3-interceptor-1.2.jar 53255 <===[JENKINS REMOTING CAPACITY]===>channel started log4j:WARN No appenders could be found for logger (org.apache.commons.beanutils.converters.BooleanConverter). log4j:WARN Please initialize the log4j system properly. Executing Maven: -B -f /var/lib/jenkins/jobs/glusterfs-hadoop/workspace/pom.xml install [INFO] Scanning for projects... [WARNING] [WARNING] Some problems were encountered while building the effective model for org.apache.hadoop.fs.glusterfs:glusterfs:jar:0.20.2-0.1 [WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but found duplicate declaration of plugin org.apache.maven.plugins:maven-compiler-plugin @ line 55, column 15 [WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-surefire-plugin is missing. @ line 24, column 15 [WARNING] [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build. [WARNING] [WARNING] For this reason, future Maven versions might no longer support building such malformed projects. [WARNING] [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building glusterfs 0.20.2-0.1 [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ glusterfs --- [debug] execute contextualize [WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent! [INFO] skip non existing resourceDirectory /var/lib/jenkins/jobs/glusterfs-hadoop/workspace/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ glusterfs --- [INFO] Nothing to compile - all classes are up to date [INFO] [debug] execute contextualize [WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent! [INFO] skip non existing resourceDirectory /var/lib/jenkins/jobs/glusterfs-hadoop/workspace/src/test/resources [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ glusterfs --- [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ glusterfs --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-surefire-plugin:2.10:test (default-test) @ glusterfs --- [INFO] Surefire report directory: /var/lib/jenkins/jobs/glusterfs-hadoop/workspace/target/surefire-reports ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.gluster.test.AppTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.068 sec Running org.gluster.test.TestConfiguration Configuration: core-default.xml, core-site.xml, file:/var/lib/jenkins/jobs/glusterfs-hadoop/workspace/conf/core-site.xml glusterfs://RHS-1:9000 , HadoopVol Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.257 sec Running org.gluster.test.TestFileInfo Confirming -- user.name(jenkins)=owner(jenkins) 0 drwxr-xr-x 1 2 2 root 3 root 4 8198 5 Feb 6 21 7 20:29 8 ./ Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.062 sec Running org.gluster.test.TestGlusterFuseInputStream Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec Running org.gluster.test.TestGluster WARNING: HOST NOT DEFINED IN ENVIRONMENT! See README WARNING: HOST NOT DEFINED IN ENVIRONMENT! See README Testing against host=ip-10-62-67-242 Testing against volume=HadoopVol Confirmed that configuration properties from gluster were found , now creating dirs Now initializing GlusterFS ! server ip-10-62-67-242 Initializing GlusterFS Running: mount -t glusterfs ip-10-62-67-242:/HadoopVol /tmp/gluster/mount Initialize: Failed to mount GlusterFS - hopefully its already mounted Directory /tmp/gluster/mount/testDirs1 already exist Directory /tmp/gluster/mount/dir.1 already exist Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.58 sec Results : Tests run: 9, Failures: 0, Errors: 0, Skipped: 0 [JENKINS] Recording test results [INFO] [INFO] --- maven-jar-plugin:2.3.2:jar (default-jar) @ glusterfs --- [INFO] [INFO] --- maven-install-plugin:2.3.1:install (default-install) @ glusterfs --- [INFO] Installing /var/lib/jenkins/jobs/glusterfs-hadoop/workspace/target/glusterfs-0.20.2-0.1.jar to /var/lib/jenkins/.m2/repository/org/apache/hadoop/fs/glusterfs/glusterfs/0.20.2-0.1/glusterfs-0.20.2-0.1.jar [INFO] Installing /var/lib/jenkins/jobs/glusterfs-hadoop/workspace/pom.xml to /var/lib/jenkins/.m2/repository/org/apache/hadoop/fs/glusterfs/glusterfs/0.20.2-0.1/glusterfs-0.20.2-0.1.pom [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 21.045s [INFO] Finished at: Wed Feb 27 20:25:26 EST 2013 [INFO] Final Memory: 7M/25M [INFO] ------------------------------------------------------------------------ [JENKINS] Archiving /var/lib/jenkins/jobs/glusterfs-hadoop/workspace/pom.xml to /var/lib/jenkins/jobs/glusterfs-hadoop/modules/org.apache.hadoop.fs.glusterfs$glusterfs/builds/2013-02-27_20-24-57/archive/org.apache.hadoop.fs.glusterfs/glusterfs/0.20.2-0.1/glusterfs-0.20.2-0.1.pom [JENKINS] Archiving /var/lib/jenkins/jobs/glusterfs-hadoop/workspace/target/glusterfs-0.20.2-0.1.jar to /var/lib/jenkins/jobs/glusterfs-hadoop/modules/org.apache.hadoop.fs.glusterfs$glusterfs/builds/2013-02-27_20-24-57/archive/org.apache.hadoop.fs.glusterfs/glusterfs/0.20.2-0.1/glusterfs-0.20.2-0.1.jar Waiting for Jenkins to finish collecting data Sending e-mails to: jvyas bchilds channel stopped Sending e-mails to: jvyas bchilds Finished: SUCCESS
Do we have the post-commit hooks working from Jenkins? Once we have a full closed Continual Integration loop that can kick off a Jenkins build after a GIT commit then we should close this ticket. I think we should open a separate ticket for having to run the build as root to get around the gluster FUSE mount authentication requirements.
looking into it there is a problem with the post receive hook actually im on this now
Post commit builds now working. The problem was that polling of git hub scm was turned off.