Bug 765333 - (GLUSTER-3601) pi estimator job failed with 2*3 striped-replicated set-up
pi estimator job failed with 2*3 striped-replicated set-up
Status: CLOSED EOL
Product: GlusterFS
Classification: Community
Component: HDFS (Show other bugs)
pre-release
x86_64 Linux
medium Severity medium
: ---
: ---
Assigned To: Steve Watt
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2011-09-21 07:36 EDT by M S Vishwanath Bhat
Modified: 2016-05-31 21:57 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-10-22 11:40:20 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description M S Vishwanath Bhat 2011-09-21 04:51:09 EDT
It happened only with quick-slave-io ON. With quick-slave-io off it's not happening. Also I observed that, when quick-slave-io is ON, slaves did not mount the gluster volume.
Comment 1 M S Vishwanath Bhat 2011-09-21 07:36:55 EDT
Created 2*3 striped-replicated gluster volume and ran pi estimator. It failed with following trace. It happened in ubuntu cluster but times are in sync. 

Removing the /tmp directory didn;t help either...

11/09/21 17:03:45 WARN mapred.JobClient: Error reading task outputhttp://ubuntu7.gluster.com:50060/tasklog?plaintext=true&taskid=attempt_201109211612_0003_r_000001_1&filter=stdout
11/09/21 17:03:45 WARN mapred.JobClient: Error reading task outputhttp://ubuntu7.gluster.com:50060/tasklog?plaintext=true&taskid=attempt_201109211612_0003_r_000001_1&filter=stderr
11/09/21 17:03:49 INFO mapred.JobClient: Task Id : attempt_201109211612_0003_m_000100_2, Status : FAILED
Error initializing attempt_201109211612_0003_m_000100_2:
java.io.FileNotFoundException: File /mnt/glusterfs/tmp/hadoop-root/mapred/system/job_201109211612_0003/job.xml does not exist.
        at org.apache.hadoop.fs.glusterfs.GlusterFileSystem.getFileStatus(GlusterFileSystem.java:276)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:192)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)
        at org.apache.hadoop.fs.glusterfs.GlusterFileSystem.copyToLocalFile(GlusterFileSystem.java:481)
        at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1197)
        at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:771)
        at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1664)
        at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:97)
        at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1629)

11/09/21 17:03:49 WARN mapred.JobClient: Error reading task outputhttp://ubuntu9.gluster.com:50060/tasklog?plaintext=true&taskid=attempt_201109211612_0003_m_000100_2&filter=stdout
11/09/21 17:03:49 WARN mapred.JobClient: Error reading task outputhttp://ubuntu9.gluster.com:50060/tasklog?plaintext=true&taskid=attempt_201109211612_0003_m_000100_2&filter=stderr
11/09/21 17:03:52 INFO mapred.JobClient: Task Id : attempt_201109211612_0003_r_000001_2, Status : FAILED
Error initializing attempt_201109211612_0003_r_000001_2:
java.io.FileNotFoundException: File /mnt/glusterfs/tmp/hadoop-root/mapred/system/job_201109211612_0003/job.xml does not exist.
        at org.apache.hadoop.fs.glusterfs.GlusterFileSystem.getFileStatus(GlusterFileSystem.java:276)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:192)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)
        at org.apache.hadoop.fs.glusterfs.GlusterFileSystem.copyToLocalFile(GlusterFileSystem.java:481)
        at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1197)
        at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:771)
        at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1664)
        at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:97)
        at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1629)

11/09/21 17:03:52 WARN mapred.JobClient: Error reading task outputhttp://ubuntu9.gluster.com:50060/tasklog?plaintext=true&taskid=attempt_201109211612_0003_r_000001_2&filter=stdout
11/09/21 17:03:52 WARN mapred.JobClient: Error reading task outputhttp://ubuntu9.gluster.com:50060/tasklog?plaintext=true&taskid=attempt_201109211612_0003_r_000001_2&filter=stderr
11/09/21 17:03:56 INFO mapred.JobClient: Job complete: job_201109211612_0003
11/09/21 17:03:56 INFO mapred.JobClient: Counters: 0
java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
        at org.apache.hadoop.examples.PiEstimator.estimate(PiEstimator.java:297)
        at org.apache.hadoop.examples.PiEstimator.run(PiEstimator.java:342)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.examples.PiEstimator.main(PiEstimator.java:351)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
        at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Comment 2 Kaleb KEITHLEY 2015-10-22 11:40:20 EDT
pre-release version is ambiguous and about to be removed as a choice.

If you believe this is still a bug, please change the status back to NEW and choose the appropriate, applicable version for it.

Note You need to log in before you can comment on or make changes to this bug.