Bug 1023546
| Summary: | bin/hadoop -job history all <job output directory> fails with Caused by: java.io.IOException: History directory /mnt/glusterfs/out-dir/_logs/historydoes not exist | ||
|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Diane Feddema <dfeddema> |
| Component: | gluster-hadoop | Assignee: | Bradley Childs <bchilds> |
| Status: | CLOSED EOL | QA Contact: | hcfs-gluster-bugs |
| Severity: | unspecified | Docs Contact: | |
| Priority: | high | ||
| Version: | mainline | CC: | bugs, chrisw, eboyd, matt, rhs-bugs, vbellur |
| Target Milestone: | --- | Keywords: | Triaged |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-09-18 08:40:43 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Per 11/13 bug triage meeting, re-assigning to bchilds and mark high. gluster-hadoop project is not being actively maintained, and the team is not planning to contribute further on to it. |
Description of problem: Cannot display job history output on RHS. [root@gprfs001 hadoop-1.2.0.1.3.2.0-110]# bin/hadoop job -history all /mnt/glusterfs/out-dir 13/10/23 20:38:16 INFO glusterfs.GlusterFileSystemCRC: Initializing gluster volume.. 13/10/23 20:38:16 INFO glusterfs.GlusterFileSystem: Configuring GlusterFS 13/10/23 20:38:16 INFO glusterfs.GlusterFileSystem: Initializing GlusterFS, CRC disabled. 13/10/23 20:38:16 INFO glusterfs.GlusterFileSystem: Configuring GlusterFS 13/10/23 20:38:16 INFO glusterfs.GlusterFileSystemCRC: Initializing gluster volume.. Exception in thread "main" java.io.IOException: Not able to initialize History viewer at org.apache.hadoop.mapred.HistoryViewer.<init>(HistoryViewer.java:95) at org.apache.hadoop.mapred.JobClient.viewHistory(JobClient.java:1917) at org.apache.hadoop.mapred.JobClient.run(JobClient.java:1866) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.mapred.JobClient.main(JobClient.java:2123) Caused by: java.io.IOException: History directory /mnt/glusterfs/out-dir/_logs/historydoes not exist at org.apache.hadoop.mapred.HistoryViewer.<init>(HistoryViewer.java:76) ... 5 more [root@gprfs001 hadoop-1.2.0.1.3.2.0-110]# ls /mnt/glusterfs/out-dir/_logs/history job_201310231749_0003_1382566782215_root_TeraSort_default_* job_201310231749_0003_conf.xml Version-Release number of selected component (if applicable): How reproducible: 100% Steps to Reproduce: 1. Run Teragen "bin/hadoop jar hadoop-examples-*.jar teragen -Dmapred.map.tasks=32 1000000000 in-dir" 2. Run TeraSort "bin/hadoop jar hadoop-examples-*.jar terasort in-dir out-dir" 3. Run "bin/hadoop job -history all /mnt/glusterfs/out-dir" Actual results: [root@gprfs001 hadoop-1.2.0.1.3.2.0-110]# bin/hadoop job -history all /mnt/glusterfs/out-dir 13/10/23 20:38:16 INFO glusterfs.GlusterFileSystemCRC: Initializing gluster volume.. 13/10/23 20:38:16 INFO glusterfs.GlusterFileSystem: Configuring GlusterFS 13/10/23 20:38:16 INFO glusterfs.GlusterFileSystem: Initializing GlusterFS, CRC disabled. 13/10/23 20:38:16 INFO glusterfs.GlusterFileSystem: Configuring GlusterFS 13/10/23 20:38:16 INFO glusterfs.GlusterFileSystemCRC: Initializing gluster volume.. Exception in thread "main" java.io.IOException: Not able to initialize History viewer at org.apache.hadoop.mapred.HistoryViewer.<init>(HistoryViewer.java:95) at org.apache.hadoop.mapred.JobClient.viewHistory(JobClient.java:1917) at org.apache.hadoop.mapred.JobClient.run(JobClient.java:1866) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.mapred.JobClient.main(JobClient.java:2123) Caused by: java.io.IOException: History directory /mnt/glusterfs/out-dir/_logs/historydoes not exist at org.apache.hadoop.mapred.HistoryViewer.<init>(HistoryViewer.java:76) Expected results: Hadoop job: 0002_1382667767787_root ===================================== Job tracker host name: job job tracker start time: Tue May 18 19:30:41 EDT 1976 User: root JobName: TeraSort JobConf: glusterfs:/tmp/hadoop/mapred/staging/root/.staging/job_201310241156_0002/job.xml Submitted At: 24-Oct-2013 22:22:47 Launched At: 24-Oct-2013 22:22:48 (1sec) Finished At: 24-Oct-2013 22:38:04 (15mins, 15sec) Status: SUCCESS Counters: |Group Name |Counter name |Map Value |Reduce Value|Total Value| --------------------------------------------------------------------------------------- |Job Counters |Launched reduce tasks |0 |0 |24 |Job Counters |SLOTS_MILLIS_MAPS |0 |0 |13,029,335 |Job Counters |Total time spent by all reduces waiting after reserving slots (ms)|0 |0 |0 |Job Counters |Total time spent by all maps waiting after reserving slots (ms)|0 |0 |0 |Job Counters |Rack-local map tasks |0 |0 |213 |Job Counters |Launched map tasks |0 |0 |1,504 |Job Counters |Data-local map tasks |0 |0 |1,291 |Job Counters |SLOTS_MILLIS_REDUCES |0 |0 |18,011,982 |File Input Format Counters |Bytes Read |100,004,520,512|0 |100,004,520,512 |File Output Format Counters |Bytes Written |0 |100,000,000,000|100,000,000,000 |FileSystemCounters |FILE_BYTES_READ |102,212,898,824|102,000,001,440|204,212,900,264 |FileSystemCounters |GLUSTERFS_BYTES_READ |100,010,579,371|0 |100,010,579,371 |FileSystemCounters |FILE_BYTES_WRITTEN |204,065,952,370|102,001,024,994|306,066,977,364 |FileSystemCounters |GLUSTERFS_BYTES_WRITTEN |0 |100,000,000,000|100,000,000,000 |Map-Reduce Framework |Map output materialized bytes |102,000,216,576|0 |102,000,216,576 |Map-Reduce Framework |Map input records |1,000,000,000|0 |1,000,000,000 |Map-Reduce Framework |Reduce shuffle bytes |0 |102,000,216,576|102,000,216,576 |Map-Reduce Framework |Spilled Records |2,000,000,000|1,000,000,000|3,000,000,000 |Map-Reduce Framework |Map output bytes |100,000,000,000|0 |100,000,000,000 |Map-Reduce Framework |Total committed heap usage (bytes)|1,140,915,109,888|21,650,800,640|1,162,565,910,528 |Map-Reduce Framework |CPU time spent (ms) |8,365,530 |3,913,250 |12,278,780 |Map-Reduce Framework |Map input bytes |100,000,000,000|0 |100,000,000,000 |Map-Reduce Framework |SPLIT_RAW_BYTES |121,824 |0 |121,824 |Map-Reduce Framework |Combine input records |0 |0 |0 |Map-Reduce Framework |Reduce input records |0 |1,000,000,000|1,000,000,000 |Map-Reduce Framework |Reduce input groups |0 |1,000,000,000|1,000,000,000 |Map-Reduce Framework |Combine output records |0 |0 |0 |Map-Reduce Framework |Physical memory (bytes) snapshot|374,024,769,536|24,219,992,064|398,244,761,600 |Map-Reduce Framework |Reduce output records |0 |1,000,000,000|1,000,000,000 |Map-Reduce Framework |Virtual memory (bytes) snapshot|2,431,658,258,432|38,943,023,104|2,470,601,281,536 |Map-Reduce Framework |Map output records |1,000,000,000|0 |1,000,000,000 ===================================== Task Summary ============================ Kind Total Successful Failed Killed StartTime FinishTime Setup 1 1 0 0 24-Oct-2013 22:22:49 24-Oct-2013 22:22:51 (1sec) Map 1504 1504 0 0 24-Oct-2013 22:22:51 24-Oct-2013 22:29:58 (7mins, 7sec) Reduce 24 24 0 0 24-Oct-2013 22:23:14 24-Oct-2013 22:37:59 (14mins, 45sec) Cleanup 1 1 0 0 24-Oct-2013 22:37:59 24-Oct-2013 22:38:04 (4sec) ============================ Analysis ========= Time taken by best performing map task task_201310241156_0002_m_001490: 4sec Average time taken by map tasks: 8sec Worse performing map tasks: TaskId Timetaken task_201310241156_0002_m_000170 46sec task_201310241156_0002_m_000449 45sec task_201310241156_0002_m_000177 40sec task_201310241156_0002_m_000181 34sec task_201310241156_0002_m_000182 34sec task_201310241156_0002_m_000184 32sec task_201310241156_0002_m_000185 32sec task_201310241156_0002_m_000186 32sec task_201310241156_0002_m_000189 31sec task_201310241156_0002_m_000652 30sec The last map task task_201310241156_0002_m_001489 finished at (relative to the Job launch time): 24-Oct-2013 22:29:58 (7mins, 10sec) Time taken by best performing shuffle task task_201310241156_0002_r_000021: 6mins, 45sec Average time taken by shuffle tasks: 6mins, 47sec Worse performing shuffle tasks: TaskId Timetaken task_201310241156_0002_r_000001 6mins, 50sec task_201310241156_0002_r_000015 6mins, 49sec task_201310241156_0002_r_000016 6mins, 48sec task_201310241156_0002_r_000007 6mins, 48sec task_201310241156_0002_r_000018 6mins, 48sec task_201310241156_0002_r_000012 6mins, 48sec task_201310241156_0002_r_000000 6mins, 48sec task_201310241156_0002_r_000006 6mins, 47sec task_201310241156_0002_r_000008 6mins, 47sec task_201310241156_0002_r_000011 6mins, 47sec The last shuffle task task_201310241156_0002_r_000000 finished at (relative to the Job launch time): 24-Oct-2013 22:30:02 (7mins, 13sec) Time taken by best performing reduce task task_201310241156_0002_r_000011: 4mins, 32sec Average time taken by reduce tasks: 5mins, 43sec Worse performing reduce tasks: TaskId Timetaken task_201310241156_0002_r_000004 7mins, 57sec task_201310241156_0002_r_000000 7mins, 57sec task_201310241156_0002_r_000020 7mins, 54sec task_201310241156_0002_r_000008 7mins, 51sec task_201310241156_0002_r_000012 7mins, 40sec task_201310241156_0002_r_000016 7mins, 39sec task_201310241156_0002_r_000014 5mins, 41sec task_201310241156_0002_r_000017 5mins, 31sec task_201310241156_0002_r_000018 5mins, 31sec task_201310241156_0002_r_000006 5mins, 31sec The last reduce task task_201310241156_0002_r_000000 finished at (relative to the Job launch time): 24-Oct-2013 22:37:59 (15mins, 10sec) ========= FAILED SETUP task list for 0002_1382667767787_root TaskId StartTime FinishTime Error ==================================================== KILLED SETUP task list for 0002_1382667767787_root TaskId StartTime FinishTime Error ==================================================== FAILED MAP task list for 0002_1382667767787_root TaskId StartTime FinishTime Error InputSplits ==================================================== KILLED MAP task list for 0002_1382667767787_root TaskId StartTime FinishTime Error InputSplits ==================================================== FAILED REDUCE task list for 0002_1382667767787_root TaskId StartTime FinishTime Error ==================================================== KILLED REDUCE task list for 0002_1382667767787_root TaskId StartTime FinishTime Error ==================================================== FAILED CLEANUP task list for 0002_1382667767787_root TaskId StartTime FinishTime Error ==================================================== KILLED CLEANUP task list for 0002_1382667767787_root TaskId StartTime FinishTime Error ==================================================== SUCCESS SETUP task list for 0002_1382667767787_root TaskId StartTime FinishTime Error ==================================================== task_201310241156_0002_m_001505 24-Oct-2013 22:22:48 24-Oct-2013 22:22:51 (2sec) SUCCESS MAP task list for 0002_1382667767787_root TaskId StartTime FinishTime Error InputSplits ==================================================== task_201310241156_0002_m_000000 24-Oct-2013 22:22:51 24-Oct-2013 22:23:01 (9sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000001 24-Oct-2013 22:22:51 24-Oct-2013 22:23:01 (9sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000002 24-Oct-2013 22:22:51 24-Oct-2013 22:23:01 (9sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000003 24-Oct-2013 22:22:51 24-Oct-2013 22:23:01 (9sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000004 24-Oct-2013 22:22:51 24-Oct-2013 22:23:01 (9sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000005 24-Oct-2013 22:22:51 24-Oct-2013 22:23:01 (9sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000006 24-Oct-2013 22:22:51 24-Oct-2013 22:23:01 (9sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000007 24-Oct-2013 22:22:51 24-Oct-2013 22:23:01 (9sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000008 24-Oct-2013 22:22:51 24-Oct-2013 22:23:01 (9sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000009 24-Oct-2013 22:22:51 24-Oct-2013 22:23:01 (9sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000010 24-Oct-2013 22:22:51 24-Oct-2013 22:23:01 (9sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000011 24-Oct-2013 22:22:51 24-Oct-2013 22:23:01 (9sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000012 24-Oct-2013 22:22:51 24-Oct-2013 22:23:01 (9sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000013 24-Oct-2013 22:22:51 24-Oct-2013 22:23:01 (9sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000014 24-Oct-2013 22:22:51 24-Oct-2013 22:23:01 (9sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000015 24-Oct-2013 22:22:51 24-Oct-2013 22:23:01 (9sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000016 24-Oct-2013 22:23:01 24-Oct-2013 22:23:08 (7sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000017 24-Oct-2013 22:23:01 24-Oct-2013 22:23:08 (7sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000018 24-Oct-2013 22:23:01 24-Oct-2013 22:23:08 (7sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000019 24-Oct-2013 22:23:01 24-Oct-2013 22:23:08 (7sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000020 24-Oct-2013 22:23:01 24-Oct-2013 22:23:08 (7sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000021 24-Oct-2013 22:23:01 24-Oct-2013 22:23:08 (7sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000022 24-Oct-2013 22:23:01 24-Oct-2013 22:23:08 (7sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000023 24-Oct-2013 22:23:01 24-Oct-2013 22:23:08 (7sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000024 24-Oct-2013 22:23:01 24-Oct-2013 22:23:08 (7sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000025 24-Oct-2013 22:23:01 24-Oct-2013 22:23:08 (7sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000026 24-Oct-2013 22:23:01 24-Oct-2013 22:23:09 (7sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000027 24-Oct-2013 22:23:01 24-Oct-2013 22:23:08 (7sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000028 24-Oct-2013 22:23:01 24-Oct-2013 22:23:08 (7sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000029 24-Oct-2013 22:23:01 24-Oct-2013 22:23:09 (7sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000030 24-Oct-2013 22:23:01 24-Oct-2013 22:23:08 (7sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000031 24-Oct-2013 22:23:01 24-Oct-2013 22:23:08 (7sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000032 24-Oct-2013 22:23:08 24-Oct-2013 22:23:14 (6sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000033 24-Oct-2013 22:23:08 24-Oct-2013 22:23:15 (6sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000034 24-Oct-2013 22:23:08 24-Oct-2013 22:23:14 (5sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000035 24-Oct-2013 22:23:08 24-Oct-2013 22:23:14 (6sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000036 24-Oct-2013 22:23:08 24-Oct-2013 22:23:15 (6sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000037 24-Oct-2013 22:23:08 24-Oct-2013 22:23:15 (6sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000038 24-Oct-2013 22:23:08 24-Oct-2013 22:23:15 (6sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000039 24-Oct-2013 22:23:08 24-Oct-2013 22:23:15 (6sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000040 24-Oct-2013 22:23:08 24-Oct-2013 22:23:14 (6sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000041 24-Oct-2013 22:23:08 24-Oct-2013 22:23:14 (5sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000042 24-Oct-2013 22:23:08 24-Oct-2013 22:23:14 (5sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000043 24-Oct-2013 22:23:08 24-Oct-2013 22:23:14 (5sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000044 24-Oct-2013 22:23:08 24-Oct-2013 22:23:14 (5sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000045 24-Oct-2013 22:23:08 24-Oct-2013 22:23:14 (5sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000046 24-Oct-2013 22:23:09 24-Oct-2013 22:23:15 (6sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000047 24-Oct-2013 22:23:09 24-Oct-2013 22:23:15 (6sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000048 24-Oct-2013 22:23:14 24-Oct-2013 22:23:22 (7sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000049 24-Oct-2013 22:23:14 24-Oct-2013 22:23:22 (7sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000050 24-Oct-2013 22:23:14 24-Oct-2013 22:23:21 (6sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000051 24-Oct-2013 22:23:14 24-Oct-2013 22:23:22 (7sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000052 24-Oct-2013 22:23:14 24-Oct-2013 22:23:22 (7sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000053 24-Oct-2013 22:23:14 24-Oct-2013 22:23:23 (8sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000054 24-Oct-2013 22:23:15 24-Oct-2013 22:23:21 (6sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000055 24-Oct-2013 22:23:15 24-Oct-2013 22:23:22 (7sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000056 24-Oct-2013 22:23:15 24-Oct-2013 22:23:24 (9sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com task_201310241156_0002_m_000057 24-Oct-2013 22:23:15 24-Oct-2013 22:23:23 (8sec) /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com etc etc Additional info: