Bug 765333 (GLUSTER-3601)

Summary: pi estimator job failed with 2*3 striped-replicated set-up
Product: [Community] GlusterFS Reporter: M S Vishwanath Bhat <vbhat>
Component: HDFSAssignee: Steve Watt <swatt>
Status: CLOSED EOL QA Contact:
Severity: medium Docs Contact:
Priority: medium    
Version: pre-releaseCC: bugs, gluster-bugs, mzywusko
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-10-22 15:40:20 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description M S Vishwanath Bhat 2011-09-21 08:51:09 UTC 
It happened only with quick-slave-io ON. With quick-slave-io off it's not happening. Also I observed that, when quick-slave-io is ON, slaves did not mount the gluster volume.
Comment 1 M S Vishwanath Bhat 2011-09-21 11:36:55 UTC 
Created 2*3 striped-replicated gluster volume and ran pi estimator. It failed with following trace. It happened in ubuntu cluster but times are in sync. 

Removing the /tmp directory didn;t help either...

11/09/21 17:03:45 WARN mapred.JobClient: Error reading task outputhttp://ubuntu7.gluster.com:50060/tasklog?plaintext=true&taskid=attempt_201109211612_0003_r_000001_1&filter=stdout
11/09/21 17:03:45 WARN mapred.JobClient: Error reading task outputhttp://ubuntu7.gluster.com:50060/tasklog?plaintext=true&taskid=attempt_201109211612_0003_r_000001_1&filter=stderr
11/09/21 17:03:49 INFO mapred.JobClient: Task Id : attempt_201109211612_0003_m_000100_2, Status : FAILED
Error initializing attempt_201109211612_0003_m_000100_2:
java.io.FileNotFoundException: File /mnt/glusterfs/tmp/hadoop-root/mapred/system/job_201109211612_0003/job.xml does not exist.
        at org.apache.hadoop.fs.glusterfs.GlusterFileSystem.getFileStatus(GlusterFileSystem.java:276)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:192)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)
        at org.apache.hadoop.fs.glusterfs.GlusterFileSystem.copyToLocalFile(GlusterFileSystem.java:481)
        at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1197)
        at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:771)
        at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1664)
        at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:97)
        at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1629)

11/09/21 17:03:49 WARN mapred.JobClient: Error reading task outputhttp://ubuntu9.gluster.com:50060/tasklog?plaintext=true&taskid=attempt_201109211612_0003_m_000100_2&filter=stdout
11/09/21 17:03:49 WARN mapred.JobClient: Error reading task outputhttp://ubuntu9.gluster.com:50060/tasklog?plaintext=true&taskid=attempt_201109211612_0003_m_000100_2&filter=stderr
11/09/21 17:03:52 INFO mapred.JobClient: Task Id : attempt_201109211612_0003_r_000001_2, Status : FAILED
Error initializing attempt_201109211612_0003_r_000001_2:
java.io.FileNotFoundException: File /mnt/glusterfs/tmp/hadoop-root/mapred/system/job_201109211612_0003/job.xml does not exist.
        at org.apache.hadoop.fs.glusterfs.GlusterFileSystem.getFileStatus(GlusterFileSystem.java:276)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:192)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)
        at org.apache.hadoop.fs.glusterfs.GlusterFileSystem.copyToLocalFile(GlusterFileSystem.java:481)
        at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1197)
        at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:771)
        at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1664)
        at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:97)
        at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1629)

11/09/21 17:03:52 WARN mapred.JobClient: Error reading task outputhttp://ubuntu9.gluster.com:50060/tasklog?plaintext=true&taskid=attempt_201109211612_0003_r_000001_2&filter=stdout
11/09/21 17:03:52 WARN mapred.JobClient: Error reading task outputhttp://ubuntu9.gluster.com:50060/tasklog?plaintext=true&taskid=attempt_201109211612_0003_r_000001_2&filter=stderr
11/09/21 17:03:56 INFO mapred.JobClient: Job complete: job_201109211612_0003
11/09/21 17:03:56 INFO mapred.JobClient: Counters: 0
java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
        at org.apache.hadoop.examples.PiEstimator.estimate(PiEstimator.java:297)
        at org.apache.hadoop.examples.PiEstimator.run(PiEstimator.java:342)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.examples.PiEstimator.main(PiEstimator.java:351)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
        at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Comment 2 Kaleb KEITHLEY 2015-10-22 15:40:20 UTC 
pre-release version is ambiguous and about to be removed as a choice.

If you believe this is still a bug, please change the status back to NEW and choose the appropriate, applicable version for it.