Description of problem:

  tests/bugs/bug-865825.t is failing fairly consistently in Rackspace.
  This is due to the test only waiting 3 seconds for the self healing
  daemon to start, which isn't always long enough.

  With some echo statements added so we can see what's going on,
  this shows the failure case ("not ok 17"):

    # tests/bugs/bug-865825.t
    1..21
    TEST glusterd
    ok 1
    TEST pidof glusterd
    ok 2
    TEST gluster --mode=script --wignore volume info
    No volumes present
    ok 3
    TEST gluster --mode=script --wignore volume create patchy replica 3 jc0.cloud.gluster.org:/d/backends/patchy-0 jc0.cloud.gluster.org:/d/backends/patchy-1 jc0.cloud.gluster.org:/d/backends/patchy-2
    ok 4
    EXPECT patchy volinfo_field patchy Volume Name
    ok 5
    EXPECT Created volinfo_field patchy Status
    ok 6
    TEST gluster --mode=script --wignore volume set patchy cluster.background-self-heal-count 0
    ok 7
    TEST gluster --mode=script --wignore volume set patchy performance.io-cache off
    ok 8
    TEST gluster --mode=script --wignore volume set patchy performance.quick-read off
    ok 9
    TEST gluster --mode=script --wignore volume set patchy performance.write-behind off
    ok 10
    TEST gluster --mode=script --wignore volume set patchy performance.stat-prefetch off
    ok 11
    TEST gluster --mode=script --wignore volume set patchy cluster.self-heal-daemon off
    ok 12
    TEST gluster --mode=script --wignore volume start patchy
    ok 13
    EXPECT Started volinfo_field patchy Status
    ok 14
    TEST glusterfs --volfile-server=jc0.cloud.gluster.org --volfile-id=patchy /mnt/glusterfs/0
    ok 15
    TEST umount /mnt/glusterfs/0
    ok 16
    setfattr: /d/backends/patchy-1/a_file: No such attribute
    volume set: success
    Self-heal daemon is not running. Check self-heal daemon log file. 
    EXPECT_WITHIN 30 test_data cat /d/backends/patchy-2/a_file
    not ok 17
    TEST gluster --mode=script --wignore volume stop patchy
    ok 18
    EXPECT Stopped volinfo_field patchy Status
    ok 19
    TEST gluster --mode=script --wignore volume delete patchy
    ok 20
    TEST ! gluster --mode=script --wignore volume info patchy
    ok 21

  This shows the success case, with all tests passing:

    # tests/bugs/bug-865825.t
    1..21
    TEST glusterd
    ok 1
    TEST pidof glusterd
    ok 2
    TEST gluster --mode=script --wignore volume info
    No volumes present
    ok 3
    TEST gluster --mode=script --wignore volume create patchy replica 3 jc0.cloud.gluster.org:/d/backends/patchy-0 jc0.cloud.gluster.org:/d/backends/patchy-1 jc0.cloud.gluster.org:/d/backends/patchy-2
    ok 4
    EXPECT patchy volinfo_field patchy Volume Name
    ok 5
    EXPECT Created volinfo_field patchy Status
    ok 6
    TEST gluster --mode=script --wignore volume set patchy cluster.background-self-heal-count 0
    ok 7
    TEST gluster --mode=script --wignore volume set patchy performance.io-cache off
    ok 8
    TEST gluster --mode=script --wignore volume set patchy performance.quick-read off
    ok 9
    TEST gluster --mode=script --wignore volume set patchy performance.write-behind off
    ok 10
    TEST gluster --mode=script --wignore volume set patchy performance.stat-prefetch off
    ok 11
    TEST gluster --mode=script --wignore volume set patchy cluster.self-heal-daemon off
    ok 12
    TEST gluster --mode=script --wignore volume start patchy
    ok 13
    EXPECT Started volinfo_field patchy Status
    ok 14
    TEST glusterfs --volfile-server=jc0.cloud.gluster.org --volfile-id=patchy /mnt/glusterfs/0
    ok 15
    TEST umount /mnt/glusterfs/0
    ok 16
    setfattr: /d/backends/patchy-1/a_file: No such attribute
    volume set: success
    Launching heal operation to perform full self heal on volume patchy has been successful 
    Use heal info commands to check status
    EXPECT_WITHIN 30 test_data cat /d/backends/patchy-2/a_file
    ok 17
    TEST gluster --mode=script --wignore volume stop patchy
    ok 18
    EXPECT Stopped volinfo_field patchy Status
    ok 19
    TEST gluster --mode=script --wignore volume delete patchy
    ok 20
    TEST ! gluster --mode=script --wignore volume info patchy
    ok 21

  In the test source code, there is a "sleep 3" to wait for the
  self-heal daemon just before the error.  Increasing the delay
  to longer than 3 seconds seems to work, with the test no longer
  failing in Rackspace.

  I'll submit a patch for review in Gerrit shortly, with a change
  to 10 seconds (just to be safe).


Version-Release number of selected component (if applicable):

  Upstream git master, as of Sat 5th Apr 2014
  commit d8dd4049143c191cea451bade470b906c67dbbe0


How reproducible:

  Frequently failing, but not every time. :/