Bug 851509 - Multiple failures were detected while executing posix sanity tests on POSIX compliant FS storage type.
Summary: Multiple failures were detected while executing posix sanity tests on POSIX c...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: glusterfs
Version: 2.0
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: shishir gowda
QA Contact: Sudhir D
URL:
Whiteboard:
: 858470 (view as bug list)
Depends On:
Blocks: 858470
TreeView+ depends on / blocked
 
Reported: 2012-08-24 11:55 UTC by Gowrishankar Rajaiyan
Modified: 2013-12-09 01:33 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 858470 (view as bug list)
Environment:
Last Closed: 2012-09-24 07:28:47 UTC
Embargoed:

Attachments (Terms of Use)

Description Gowrishankar Rajaiyan 2012-08-24 11:55:52 UTC 
Description of problem: This was detected during RHEV3.1-RHS2.0+ testing. Multiple failures were detected as part of posix-test sanity run.


Version-Release number of selected component (if applicable):
glusterfs-server-3.3.0rhs-26.el6rhs.x86_64

How reproducible:
No exact reproducible steps, since these failures are detected randomly.

Steps to Reproduce: Not sure the steps are relevant but still mentioning it here.
1. Setup RHEV environment
2. Create Datacenter with Storage as gluster mount (POSIX compliant FS).
3. Create a virtual machine on this storage.
4. Execute posix_compliance sanity tests on the mount.
  
Actual results:
1. 
Test Summary Report
-------------------
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
/opt/qa/tools/posix-testsuite/tests/link/00.t    (Wstat: 0 Tests: 82 Failed: 1)
  Failed test:  56
/opt/qa/tools/posix-testsuite/tests/mkfifo/00.t  (Wstat: 0 Tests: 36 Failed: 5)
  Failed tests:  30-34
/opt/qa/tools/posix-testsuite/tests/open/00.t    (Wstat: 0 Tests: 47 Failed: 5)
  Failed tests:  30-34
/opt/qa/tools/posix-testsuite/tests/rename/00.t  (Wstat: 0 Tests: 79 Failed: 2)
  Failed tests:  49, 57
/opt/qa/tools/posix-testsuite/tests/unlink/00.t  (Wstat: 0 Tests: 55 Failed: 2)
  Failed tests:  47-48
Files=185, Tests=1979, 84 wallclock secs ( 0.88 usr  0.31 sys +  9.12 cusr  9.00 csys = 19.31 CPU)
Result: FAIL
end: 21:26:23
removed posix compliance directories
1
Total 1 tests were successful
Switching over to the previous working directory
Removing /rhev/data-center/mnt/rhs006.lab.eng.blr.redhat.com:_dist-replica-3-volume/run13823/
[root@test system_light]#

2. 
Test Summary Report
-------------------
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
/opt/qa/tools/posix-testsuite/tests/link/00.t    (Wstat: 0 Tests: 82 Failed: 4)
  Failed tests:  57-58, 64-65
/opt/qa/tools/posix-testsuite/tests/rename/00.t  (Wstat: 0 Tests: 79 Failed: 2)
  Failed tests:  49, 57
/opt/qa/tools/posix-testsuite/tests/symlink/00.t (Wstat: 0 Tests: 14 Failed: 2)
  Failed tests:  11-12
/opt/qa/tools/posix-testsuite/tests/unlink/00.t  (Wstat: 0 Tests: 55 Failed: 6)
  Failed tests:  35-36, 41-42, 47-48
Files=185, Tests=1979, 85 wallclock secs ( 0.88 usr  0.29 sys +  9.30 cusr  8.95 csys = 19.42 CPU)
Result: FAIL
end: 21:33:03
removed posix compliance directories
1
Total 1 tests were successful
Switching over to the previous working directory
Removing /rhev/data-center/mnt/rhs006.lab.eng.blr.redhat.com:_dist-replica-3-volume/run29341/
[root@test system_light]# 


Expected results:
No failures.

Additional info: Thanks to Anush Shetty in helping me debug this.
Comment 1 Anush Shetty 2012-08-24 12:01:38 UTC 
Also, we didn't see this issue when the test was run directly on the native filesystem.
Comment 3 Gowrishankar Rajaiyan 2012-08-27 05:27:33 UTC 
Update:

Made sure time on all RHS nodes are in sync:
[root@pandora ~]# for i in 1 2 3 4 5 6; do ssh root@rhs00$i ntpdate clock.redhat.com; done
root@rhs001's password: 
27 Aug 04:59:50 ntpdate[32190]: adjust time server 10.5.26.10 offset 0.000200 sec
root@rhs002's password: 
27 Aug 04:59:53 ntpdate[30184]: adjust time server 10.5.26.10 offset 0.000143 sec
root@rhs003's password: 
27 Aug 04:59:57 ntpdate[26695]: adjust time server 10.5.26.10 offset 0.000150 sec
root@rhs004's password: 
27 Aug 05:00:01 ntpdate[27627]: adjust time server 10.5.26.10 offset -0.000139 sec
root@rhs005's password: 
27 Aug 05:00:05 ntpdate[22461]: adjust time server 10.5.26.10 offset -0.001713 sec
root@rhs006's password: 
27 Aug 05:00:08 ntpdate[24901]: adjust time server 10.5.26.10 offset 0.000285 sec
[root@pandora ~]# 


and executed the sanity tests:
<snip>
Test Summary Report
-------------------
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
/opt/qa/tools/posix-testsuite/tests/open/00.t    (Wstat: 0 Tests: 47 Failed: 5)
  Failed tests:  30-34
Files=185, Tests=1979, 82 wallclock secs ( 0.85 usr  0.33 sys +  9.16 cusr  8.86 csys = 19.20 CPU)
Result: FAIL
</snip>

<snip>
Test Summary Report
-------------------
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
/opt/qa/tools/posix-testsuite/tests/mkfifo/00.t  (Wstat: 0 Tests: 36 Failed: 5)
  Failed tests:  30-34
Files=185, Tests=1979, 82 wallclock secs ( 0.87 usr  0.32 sys +  9.20 cusr  9.01 csys = 19.40 CPU)
Result: FAIL
</snip>

<snip>
Test Summary Report
-------------------
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
/opt/qa/tools/posix-testsuite/tests/link/00.t    (Wstat: 0 Tests: 82 Failed: 1)
  Failed test:  56
/opt/qa/tools/posix-testsuite/tests/mkfifo/00.t  (Wstat: 0 Tests: 36 Failed: 5)
  Failed tests:  30-34
/opt/qa/tools/posix-testsuite/tests/open/00.t    (Wstat: 0 Tests: 47 Failed: 5)
  Failed tests:  30-34
/opt/qa/tools/posix-testsuite/tests/rename/00.t  (Wstat: 0 Tests: 79 Failed: 2)
  Failed tests:  49, 57
/opt/qa/tools/posix-testsuite/tests/unlink/00.t  (Wstat: 0 Tests: 55 Failed: 2)
  Failed tests:  47-48
Files=185, Tests=1979, 84 wallclock secs ( 0.88 usr  0.31 sys +  9.12 cusr  9.00 csys = 19.31 CPU)
Result: FAIL
</snip>

<snip>
Test Summary Report
-------------------
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
/opt/qa/tools/posix-testsuite/tests/link/00.t    (Wstat: 0 Tests: 82 Failed: 4)
  Failed tests:  57-58, 64-65
/opt/qa/tools/posix-testsuite/tests/rename/00.t  (Wstat: 0 Tests: 79 Failed: 2)
  Failed tests:  49, 57
/opt/qa/tools/posix-testsuite/tests/symlink/00.t (Wstat: 0 Tests: 14 Failed: 2)
  Failed tests:  11-12
/opt/qa/tools/posix-testsuite/tests/unlink/00.t  (Wstat: 0 Tests: 55 Failed: 6)
  Failed tests:  35-36, 41-42, 47-48
Files=185, Tests=1979, 85 wallclock secs ( 0.88 usr  0.29 sys +  9.30 cusr  8.95 csys = 19.42 CPU)
Result: FAIL
</snip>
Comment 4 Gowrishankar Rajaiyan 2012-08-27 08:17:32 UTC 
[root@rhs001 ~]# service ntpd status
ntpd (pid  1508) is running...
[root@rhs001 ~]#

[root@rhs001 ~]# cat /etc/ntp.conf | grep clock.redhat.com
server clock.redhat.com
[root@rhs001 ~]# 

[root@rhs001 ~]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*clock1.redhat.c .CDMA.           1 u   46  128  377  293.025    0.029  53.349
[root@rhs001 ~]#

[root@pandora system_light]# cat run4.log | grep -i fail
Failed 6/171 subtests 
Failed 5/47 subtests 
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
/opt/qa/tools/posix-testsuite/tests/open/00.t    (Wstat: 0 Tests: 47 Failed: 5)
  Failed tests:  30-34
Result: FAIL
Failed 6/171 subtests 
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
Result: FAIL
Failed 6/171 subtests 
Failed 5/36 subtests 
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
/opt/qa/tools/posix-testsuite/tests/mkfifo/00.t  (Wstat: 0 Tests: 36 Failed: 5)
  Failed tests:  30-34
Result: FAIL
Failed 6/171 subtests 
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
Result: FAIL
Failed 6/171 subtests 
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
Result: FAIL
Failed 6/171 subtests 
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
Result: FAIL
Failed 6/171 subtests 
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
Result: FAIL
Failed 6/171 subtests 
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
Result: FAIL
Failed 6/171 subtests 
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
Result: FAIL
Failed 6/171 subtests 
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
Result: FAIL
Failed 6/171 subtests 
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
Result: FAIL
Failed 6/171 subtests 
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
Result: FAIL
Failed 6/171 subtests 
Failed 1/82 subtests 
Failed 5/36 subtests 
Failed 5/47 subtests 
Failed 2/79 subtests 
Failed 2/55 subtests 
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
/opt/qa/tools/posix-testsuite/tests/link/00.t    (Wstat: 0 Tests: 82 Failed: 1)
  Failed test:  56
/opt/qa/tools/posix-testsuite/tests/mkfifo/00.t  (Wstat: 0 Tests: 36 Failed: 5)
  Failed tests:  30-34
/opt/qa/tools/posix-testsuite/tests/open/00.t    (Wstat: 0 Tests: 47 Failed: 5)
  Failed tests:  30-34
/opt/qa/tools/posix-testsuite/tests/rename/00.t  (Wstat: 0 Tests: 79 Failed: 2)
  Failed tests:  49, 57
/opt/qa/tools/posix-testsuite/tests/unlink/00.t  (Wstat: 0 Tests: 55 Failed: 2)
  Failed tests:  47-48
Result: FAIL
Failed 6/171 subtests 
Failed 4/82 subtests 
Failed 2/79 subtests 
Failed 2/14 subtests 
Failed 6/55 subtests 
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
/opt/qa/tools/posix-testsuite/tests/link/00.t    (Wstat: 0 Tests: 82 Failed: 4)
  Failed tests:  57-58, 64-65
/opt/qa/tools/posix-testsuite/tests/rename/00.t  (Wstat: 0 Tests: 79 Failed: 2)
  Failed tests:  49, 57
/opt/qa/tools/posix-testsuite/tests/symlink/00.t (Wstat: 0 Tests: 14 Failed: 2)
  Failed tests:  11-12
/opt/qa/tools/posix-testsuite/tests/unlink/00.t  (Wstat: 0 Tests: 55 Failed: 6)
  Failed tests:  35-36, 41-42, 47-48
Result: FAIL
Failed 6/171 subtests 
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
Result: FAIL
Failed 6/171 subtests 
Failed 5/36 subtests 
Failed 1/79 subtests 
Failed 2/55 subtests 
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
/opt/qa/tools/posix-testsuite/tests/mkfifo/00.t  (Wstat: 0 Tests: 36 Failed: 5)
  Failed tests:  30-34
/opt/qa/tools/posix-testsuite/tests/rename/00.t  (Wstat: 0 Tests: 79 Failed: 1)
  Failed test:  61
/opt/qa/tools/posix-testsuite/tests/unlink/00.t  (Wstat: 0 Tests: 55 Failed: 2)
  Failed tests:  35-36
Result: FAIL
Failed 6/171 subtests 
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
Result: FAIL
Failed 6/171 subtests 
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
Result: FAIL
Failed 6/171 subtests 
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
Result: FAIL
Failed 6/171 subtests 
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
Result: FAIL
Failed 6/171 subtests 
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
Result: FAIL
Failed 6/171 subtests 
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
Result: FAIL
Failed 6/171 subtests 
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
Result: FAIL
Failed 6/171 subtests 
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
Result: FAIL
Failed 6/171 subtests 
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
Result: FAIL
Failed 6/171 subtests 
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
Result: FAIL
Failed 6/171 subtests 
/opt/qa/tools/posix-testsuite/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
  Failed tests:  84, 88, 141, 145, 149, 153
Result: FAIL
[root@pandora system_light]#
Comment 5 Amar Tumballi 2012-09-04 09:08:22 UTC 
need to check this by disabling all the perf xlators and running the tests.
Comment 6 shishir gowda 2012-09-13 06:23:40 UTC 
Even with all the peformance xlators turned off, we seem to be hitting the test_time failures. All the above posix failures are related to comparision of time stamps of pre and post op's.

quick-read=disable
io-cache=disable
stat-prefetch=disable
read-ahead=disable
eager-locking=enable
Comment 7 shishir gowda 2012-09-24 07:28:47 UTC 
As commented above
Comment 8 shishir gowda 2012-09-24 07:29:21 UTC 
*** Bug 858470 has been marked as a duplicate of this bug. ***
Note You need to log in before you can comment on or make changes to this bug.