Bug 1023546 - bin/hadoop -job history all <job output directory> fails with Caused by: java.io.IOException: History directory /mnt/glusterfs/out-dir/_logs/historydoes not exist
bin/hadoop -job history all <job output directory> fails with Caused by: j...
Status: ASSIGNED
Product: GlusterFS
Classification: Community
Component: gluster-hadoop (Show other bugs)
mainline
Unspecified Unspecified
high Severity unspecified
: ---
: ---
Assigned To: Bradley Childs
hcfs-gluster-bugs
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-25 13:42 EDT by Diane Feddema
Modified: 2017-01-26 21:56 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Diane Feddema 2013-10-25 13:42:41 EDT
Description of problem:
Cannot display job history output on RHS. 

[root@gprfs001 hadoop-1.2.0.1.3.2.0-110]# bin/hadoop job -history all /mnt/glusterfs/out-dir
13/10/23 20:38:16 INFO glusterfs.GlusterFileSystemCRC: Initializing gluster volume..
13/10/23 20:38:16 INFO glusterfs.GlusterFileSystem: Configuring GlusterFS
13/10/23 20:38:16 INFO glusterfs.GlusterFileSystem: Initializing GlusterFS,  CRC disabled.
13/10/23 20:38:16 INFO glusterfs.GlusterFileSystem: Configuring GlusterFS
13/10/23 20:38:16 INFO glusterfs.GlusterFileSystemCRC: Initializing gluster volume..
Exception in thread "main" java.io.IOException: Not able to initialize History viewer
    at org.apache.hadoop.mapred.HistoryViewer.<init>(HistoryViewer.java:95)
    at org.apache.hadoop.mapred.JobClient.viewHistory(JobClient.java:1917)
    at org.apache.hadoop.mapred.JobClient.run(JobClient.java:1866)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
    at org.apache.hadoop.mapred.JobClient.main(JobClient.java:2123)
Caused by: java.io.IOException: History directory /mnt/glusterfs/out-dir/_logs/historydoes not exist
    at org.apache.hadoop.mapred.HistoryViewer.<init>(HistoryViewer.java:76)
    ... 5 more
[root@gprfs001 hadoop-1.2.0.1.3.2.0-110]# ls /mnt/glusterfs/out-dir/_logs/history
job_201310231749_0003_1382566782215_root_TeraSort_default_*  job_201310231749_0003_conf.xml


Version-Release number of selected component (if applicable):


How reproducible:
100%

Steps to Reproduce:
1. Run Teragen "bin/hadoop jar hadoop-examples-*.jar teragen -Dmapred.map.tasks=32 1000000000 in-dir"
2. Run TeraSort "bin/hadoop jar hadoop-examples-*.jar terasort in-dir out-dir"
3. Run "bin/hadoop job -history all /mnt/glusterfs/out-dir"

Actual results:
[root@gprfs001 hadoop-1.2.0.1.3.2.0-110]# bin/hadoop job -history all /mnt/glusterfs/out-dir
13/10/23 20:38:16 INFO glusterfs.GlusterFileSystemCRC: Initializing gluster volume..
13/10/23 20:38:16 INFO glusterfs.GlusterFileSystem: Configuring GlusterFS
13/10/23 20:38:16 INFO glusterfs.GlusterFileSystem: Initializing GlusterFS,  CRC disabled.
13/10/23 20:38:16 INFO glusterfs.GlusterFileSystem: Configuring GlusterFS
13/10/23 20:38:16 INFO glusterfs.GlusterFileSystemCRC: Initializing gluster volume..
Exception in thread "main" java.io.IOException: Not able to initialize History viewer
    at org.apache.hadoop.mapred.HistoryViewer.<init>(HistoryViewer.java:95)
    at org.apache.hadoop.mapred.JobClient.viewHistory(JobClient.java:1917)
    at org.apache.hadoop.mapred.JobClient.run(JobClient.java:1866)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
    at org.apache.hadoop.mapred.JobClient.main(JobClient.java:2123)
Caused by: java.io.IOException: History directory /mnt/glusterfs/out-dir/_logs/historydoes not exist
    at org.apache.hadoop.mapred.HistoryViewer.<init>(HistoryViewer.java:76)

Expected results:
Hadoop job: 0002_1382667767787_root
=====================================
Job tracker host name: job
job tracker start time: Tue May 18 19:30:41 EDT 1976
User: root
JobName: TeraSort
JobConf: glusterfs:/tmp/hadoop/mapred/staging/root/.staging/job_201310241156_0002/job.xml
Submitted At: 24-Oct-2013 22:22:47
Launched At: 24-Oct-2013 22:22:48 (1sec)
Finished At: 24-Oct-2013 22:38:04 (15mins, 15sec)
Status: SUCCESS
Counters:

|Group Name                    |Counter name                  |Map Value |Reduce Value|Total Value|
---------------------------------------------------------------------------------------
|Job Counters                  |Launched reduce tasks         |0         |0         |24
|Job Counters                  |SLOTS_MILLIS_MAPS             |0         |0         |13,029,335
|Job Counters                  |Total time spent by all reduces waiting after reserving slots (ms)|0         |0         |0
|Job Counters                  |Total time spent by all maps waiting after reserving slots (ms)|0         |0         |0
|Job Counters                  |Rack-local map tasks          |0         |0         |213
|Job Counters                  |Launched map tasks            |0         |0         |1,504
|Job Counters                  |Data-local map tasks          |0         |0         |1,291
|Job Counters                  |SLOTS_MILLIS_REDUCES          |0         |0         |18,011,982
|File Input Format Counters    |Bytes Read                    |100,004,520,512|0         |100,004,520,512
|File Output Format Counters   |Bytes Written                 |0         |100,000,000,000|100,000,000,000
|FileSystemCounters            |FILE_BYTES_READ               |102,212,898,824|102,000,001,440|204,212,900,264
|FileSystemCounters            |GLUSTERFS_BYTES_READ          |100,010,579,371|0         |100,010,579,371
|FileSystemCounters            |FILE_BYTES_WRITTEN            |204,065,952,370|102,001,024,994|306,066,977,364
|FileSystemCounters            |GLUSTERFS_BYTES_WRITTEN       |0         |100,000,000,000|100,000,000,000
|Map-Reduce Framework          |Map output materialized bytes |102,000,216,576|0         |102,000,216,576
|Map-Reduce Framework          |Map input records             |1,000,000,000|0         |1,000,000,000
|Map-Reduce Framework          |Reduce shuffle bytes          |0         |102,000,216,576|102,000,216,576
|Map-Reduce Framework          |Spilled Records               |2,000,000,000|1,000,000,000|3,000,000,000
|Map-Reduce Framework          |Map output bytes              |100,000,000,000|0         |100,000,000,000
|Map-Reduce Framework          |Total committed heap usage (bytes)|1,140,915,109,888|21,650,800,640|1,162,565,910,528
|Map-Reduce Framework          |CPU time spent (ms)           |8,365,530 |3,913,250 |12,278,780
|Map-Reduce Framework          |Map input bytes               |100,000,000,000|0         |100,000,000,000
|Map-Reduce Framework          |SPLIT_RAW_BYTES               |121,824   |0         |121,824
|Map-Reduce Framework          |Combine input records         |0         |0         |0
|Map-Reduce Framework          |Reduce input records          |0         |1,000,000,000|1,000,000,000
|Map-Reduce Framework          |Reduce input groups           |0         |1,000,000,000|1,000,000,000
|Map-Reduce Framework          |Combine output records        |0         |0         |0
|Map-Reduce Framework          |Physical memory (bytes) snapshot|374,024,769,536|24,219,992,064|398,244,761,600
|Map-Reduce Framework          |Reduce output records         |0         |1,000,000,000|1,000,000,000
|Map-Reduce Framework          |Virtual memory (bytes) snapshot|2,431,658,258,432|38,943,023,104|2,470,601,281,536
|Map-Reduce Framework          |Map output records            |1,000,000,000|0         |1,000,000,000
=====================================

Task Summary
============================
Kind    Total   Successful      Failed  Killed  StartTime       FinishTime

Setup   1       1               0       0       24-Oct-2013 22:22:49    24-Oct-2013 22:22:51 (1sec)
Map     1504    1504            0       0       24-Oct-2013 22:22:51    24-Oct-2013 22:29:58 (7mins, 7sec)
Reduce  24      24              0       0       24-Oct-2013 22:23:14    24-Oct-2013 22:37:59 (14mins, 45sec)
Cleanup 1       1               0       0       24-Oct-2013 22:37:59    24-Oct-2013 22:38:04 (4sec)
============================


Analysis
=========

Time taken by best performing map task task_201310241156_0002_m_001490: 4sec
Average time taken by map tasks: 8sec
Worse performing map tasks:
TaskId          Timetaken
task_201310241156_0002_m_000170 46sec
task_201310241156_0002_m_000449 45sec
task_201310241156_0002_m_000177 40sec
task_201310241156_0002_m_000181 34sec
task_201310241156_0002_m_000182 34sec
task_201310241156_0002_m_000184 32sec
task_201310241156_0002_m_000185 32sec
task_201310241156_0002_m_000186 32sec
task_201310241156_0002_m_000189 31sec
task_201310241156_0002_m_000652 30sec
The last map task task_201310241156_0002_m_001489 finished at (relative to the Job launch time): 24-Oct-2013 22:29:58 (7mins, 10sec)

Time taken by best performing shuffle task task_201310241156_0002_r_000021: 6mins, 45sec
Average time taken by shuffle tasks: 6mins, 47sec
Worse performing shuffle tasks:
TaskId          Timetaken
task_201310241156_0002_r_000001 6mins, 50sec
task_201310241156_0002_r_000015 6mins, 49sec
task_201310241156_0002_r_000016 6mins, 48sec
task_201310241156_0002_r_000007 6mins, 48sec
task_201310241156_0002_r_000018 6mins, 48sec
task_201310241156_0002_r_000012 6mins, 48sec
task_201310241156_0002_r_000000 6mins, 48sec
task_201310241156_0002_r_000006 6mins, 47sec
task_201310241156_0002_r_000008 6mins, 47sec
task_201310241156_0002_r_000011 6mins, 47sec
The last shuffle task task_201310241156_0002_r_000000 finished at (relative to the Job launch time): 24-Oct-2013 22:30:02 (7mins, 13sec)

Time taken by best performing reduce task task_201310241156_0002_r_000011: 4mins, 32sec
Average time taken by reduce tasks: 5mins, 43sec
Worse performing reduce tasks:
TaskId          Timetaken
task_201310241156_0002_r_000004 7mins, 57sec
task_201310241156_0002_r_000000 7mins, 57sec
task_201310241156_0002_r_000020 7mins, 54sec
task_201310241156_0002_r_000008 7mins, 51sec
task_201310241156_0002_r_000012 7mins, 40sec
task_201310241156_0002_r_000016 7mins, 39sec
task_201310241156_0002_r_000014 5mins, 41sec
task_201310241156_0002_r_000017 5mins, 31sec
task_201310241156_0002_r_000018 5mins, 31sec
task_201310241156_0002_r_000006 5mins, 31sec
The last reduce task task_201310241156_0002_r_000000 finished at (relative to the Job launch time): 24-Oct-2013 22:37:59 (15mins, 10sec)
=========

FAILED SETUP task list for 0002_1382667767787_root
TaskId          StartTime       FinishTime      Error
====================================================

KILLED SETUP task list for 0002_1382667767787_root
TaskId          StartTime       FinishTime      Error
====================================================

FAILED MAP task list for 0002_1382667767787_root
TaskId          StartTime       FinishTime      Error   InputSplits
====================================================

KILLED MAP task list for 0002_1382667767787_root
TaskId          StartTime       FinishTime      Error   InputSplits
====================================================

FAILED REDUCE task list for 0002_1382667767787_root
TaskId          StartTime       FinishTime      Error
====================================================

KILLED REDUCE task list for 0002_1382667767787_root
TaskId          StartTime       FinishTime      Error
====================================================

FAILED CLEANUP task list for 0002_1382667767787_root
TaskId          StartTime       FinishTime      Error
====================================================

KILLED CLEANUP task list for 0002_1382667767787_root
TaskId          StartTime       FinishTime      Error
====================================================

SUCCESS SETUP task list for 0002_1382667767787_root
TaskId          StartTime       FinishTime      Error
====================================================
task_201310241156_0002_m_001505 24-Oct-2013 22:22:48    24-Oct-2013 22:22:51 (2sec)

SUCCESS MAP task list for 0002_1382667767787_root
TaskId          StartTime       FinishTime      Error   InputSplits
====================================================
task_201310241156_0002_m_000000 24-Oct-2013 22:22:51    24-Oct-2013 22:23:01 (9sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000001 24-Oct-2013 22:22:51    24-Oct-2013 22:23:01 (9sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000002 24-Oct-2013 22:22:51    24-Oct-2013 22:23:01 (9sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000003 24-Oct-2013 22:22:51    24-Oct-2013 22:23:01 (9sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000004 24-Oct-2013 22:22:51    24-Oct-2013 22:23:01 (9sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000005 24-Oct-2013 22:22:51    24-Oct-2013 22:23:01 (9sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000006 24-Oct-2013 22:22:51    24-Oct-2013 22:23:01 (9sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000007 24-Oct-2013 22:22:51    24-Oct-2013 22:23:01 (9sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000008 24-Oct-2013 22:22:51    24-Oct-2013 22:23:01 (9sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000009 24-Oct-2013 22:22:51    24-Oct-2013 22:23:01 (9sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000010 24-Oct-2013 22:22:51    24-Oct-2013 22:23:01 (9sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000011 24-Oct-2013 22:22:51    24-Oct-2013 22:23:01 (9sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000012 24-Oct-2013 22:22:51    24-Oct-2013 22:23:01 (9sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000013 24-Oct-2013 22:22:51    24-Oct-2013 22:23:01 (9sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000014 24-Oct-2013 22:22:51    24-Oct-2013 22:23:01 (9sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000015 24-Oct-2013 22:22:51    24-Oct-2013 22:23:01 (9sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000016 24-Oct-2013 22:23:01    24-Oct-2013 22:23:08 (7sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000017 24-Oct-2013 22:23:01    24-Oct-2013 22:23:08 (7sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000018 24-Oct-2013 22:23:01    24-Oct-2013 22:23:08 (7sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000019 24-Oct-2013 22:23:01    24-Oct-2013 22:23:08 (7sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000020 24-Oct-2013 22:23:01    24-Oct-2013 22:23:08 (7sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000021 24-Oct-2013 22:23:01    24-Oct-2013 22:23:08 (7sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000022 24-Oct-2013 22:23:01    24-Oct-2013 22:23:08 (7sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000023 24-Oct-2013 22:23:01    24-Oct-2013 22:23:08 (7sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000024 24-Oct-2013 22:23:01    24-Oct-2013 22:23:08 (7sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000025 24-Oct-2013 22:23:01    24-Oct-2013 22:23:08 (7sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000026 24-Oct-2013 22:23:01    24-Oct-2013 22:23:09 (7sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000027 24-Oct-2013 22:23:01    24-Oct-2013 22:23:08 (7sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000028 24-Oct-2013 22:23:01    24-Oct-2013 22:23:08 (7sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000029 24-Oct-2013 22:23:01    24-Oct-2013 22:23:09 (7sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000030 24-Oct-2013 22:23:01    24-Oct-2013 22:23:08 (7sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000031 24-Oct-2013 22:23:01    24-Oct-2013 22:23:08 (7sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000032 24-Oct-2013 22:23:08    24-Oct-2013 22:23:14 (6sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000033 24-Oct-2013 22:23:08    24-Oct-2013 22:23:15 (6sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000034 24-Oct-2013 22:23:08    24-Oct-2013 22:23:14 (5sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000035 24-Oct-2013 22:23:08    24-Oct-2013 22:23:14 (6sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000036 24-Oct-2013 22:23:08    24-Oct-2013 22:23:15 (6sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000037 24-Oct-2013 22:23:08    24-Oct-2013 22:23:15 (6sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000038 24-Oct-2013 22:23:08    24-Oct-2013 22:23:15 (6sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000039 24-Oct-2013 22:23:08    24-Oct-2013 22:23:15 (6sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000040 24-Oct-2013 22:23:08    24-Oct-2013 22:23:14 (6sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000041 24-Oct-2013 22:23:08    24-Oct-2013 22:23:14 (5sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000042 24-Oct-2013 22:23:08    24-Oct-2013 22:23:14 (5sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000043 24-Oct-2013 22:23:08    24-Oct-2013 22:23:14 (5sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000044 24-Oct-2013 22:23:08    24-Oct-2013 22:23:14 (5sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000045 24-Oct-2013 22:23:08    24-Oct-2013 22:23:14 (5sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000046 24-Oct-2013 22:23:09    24-Oct-2013 22:23:15 (6sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000047 24-Oct-2013 22:23:09    24-Oct-2013 22:23:15 (6sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000048 24-Oct-2013 22:23:14    24-Oct-2013 22:23:22 (7sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000049 24-Oct-2013 22:23:14    24-Oct-2013 22:23:22 (7sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000050 24-Oct-2013 22:23:14    24-Oct-2013 22:23:21 (6sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000051 24-Oct-2013 22:23:14    24-Oct-2013 22:23:22 (7sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000052 24-Oct-2013 22:23:14    24-Oct-2013 22:23:22 (7sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000053 24-Oct-2013 22:23:14    24-Oct-2013 22:23:23 (8sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000054 24-Oct-2013 22:23:15    24-Oct-2013 22:23:21 (6sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000055 24-Oct-2013 22:23:15    24-Oct-2013 22:23:22 (7sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000056 24-Oct-2013 22:23:15    24-Oct-2013 22:23:24 (9sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
task_201310241156_0002_m_000057 24-Oct-2013 22:23:15    24-Oct-2013 22:23:23 (8sec)             /default-rack/gprfs002.sbu.lab.eng.bos.redhat.com,/default-rack/gprfs001.sbu.lab.eng.bos.redhat.com
etc etc

Additional info:
Comment 2 Scott Haines 2013-11-19 14:58:40 EST
Per 11/13 bug triage meeting, re-assigning to bchilds and mark high.

Note You need to log in before you can comment on or make changes to this bug.