Bug 1332264 - [RH Ceph 2.0] on client.1 Failed 2 of 258 tests - generic/312 generic/310 [NEEDINFO]
Summary: [RH Ceph 2.0] on client.1 Failed 2 of 258 tests - generic/312 generic/310
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RBD
Version: 2.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: rc
: 3.0
Assignee: Ilya Dryomov
QA Contact: ceph-qe-bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-05-02 17:10 UTC by Vasu Kulkarni
Modified: 2022-02-21 18:06 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-07-11 14:44:01 UTC
Target Upstream Version:
bhubbard: needinfo? (vakulkar)

Attachments (Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Description Vasu Kulkarni 2016-05-02 17:10:55 UTC 
Description of problem:

2 Clients runnings xfs tests, on client.0(nodename: clara005) all tests passed but on client.1(nodename: clara008) I see 2 tests failed 

2016-04-30T23:14:35.311 INFO:tasks.rbd.client.0.clara008.stdout:	generic/198 -- src/aio-dio-regress/aiodio_sparse2 not built
2016-04-30T23:14:35.319 INFO:tasks.rbd.client.0.clara008.stdout:xfs/096       [expunged]
2016-04-30T23:14:57.590 INFO:tasks.rbd.client.0.clara008.stdout:xfs/028		[03:14:35] [03:14:57] 22s
2016-04-30T23:15:13.968 INFO:tasks.rbd.client.1.clara005.stdout:generic/310		[03:14:09] [03:15:13] 64s
2016-04-30T23:15:23.218 INFO:tasks.rbd.client.1.clara005.stdout:generic/060		[03:15:22] [03:15:23] [not run]
2016-04-30T23:15:23.244 INFO:tasks.rbd.client.1.clara005.stdout:	generic/060 -- xfs_io finsert support is missing
2016-04-30T23:15:24.312 INFO:tasks.rbd.client.1.clara005.stdout:xfs/287		[03:15:23] [03:15:24] 1s
2016-04-30T23:15:24.608 INFO:tasks.rbd.client.1.clara005.stdout:generic/184		[03:15:24] [03:15:24] 0s
2016-04-30T23:15:32.241 INFO:tasks.rbd.client.0.clara008.stdout:generic/310 64s ...	[03:14:57] [03:15:32] [failed, exit status 1] - output mismatch (see /var/lib/xfstests/results//generic/310.out.bad)
2016-04-30T23:15:32.260 INFO:tasks.rbd.client.0.clara008.stdout:    --- tests/generic/310.out	2016-05-01 01:27:12.725451314 +0000
2016-04-30T23:15:32.288 INFO:tasks.rbd.client.0.clara008.stdout:    +++ /var/lib/xfstests/results//generic/310.out.bad	2016-05-01 03:15:32.237191123 +0000
2016-04-30T23:15:32.329 INFO:tasks.rbd.client.0.clara008.stdout:    @@ -1,2 +1,2 @@
2016-04-30T23:15:32.329 INFO:tasks.rbd.client.0.clara008.stdout:     QA output created by 310
2016-04-30T23:15:32.330 INFO:tasks.rbd.client.0.clara008.stdout:    -*** done
2016-04-30T23:15:32.331 INFO:tasks.rbd.client.0.clara008.stdout:    +kernel bug detected, check dmesg for more infomation.
2016-04-30T23:15:32.332 INFO:tasks.rbd.client.0.clara008.stdout:    ...
2016-04-30T23:15:32.333 INFO:tasks.rbd.client.0.clara008.stdout:    (Run 'diff -u tests/generic/310.out /var/lib/xfstests/results//generic/310.out.bad'  to see the entire diff)
2016-04-30T23:15:33.009 INFO:tasks.rbd.client.0.clara008.stdout:generic/060		[03:15:32] [03:15:33] [not run]
2016-04-30T23:15:33.021 INFO:tasks.rbd.client.0.clara008.stdout:	generic/060 -- xfs_io finsert support is missing
2016-04-30T23:15:34.276 INFO:tasks.rbd.client.0.clara008.stdout:xfs/287 1s ...	[03:15:33] [03:15:34] 1s
2016-04-30T23:15:34.579 INFO:tasks.rbd.client.0.clara008.stdout:generic/184 1s ...	[03:15:34] [03:15:34] 0s
2016-04-30T23:15:56.053 INFO:tasks.rbd.client.1.clara005.stdout:generic/055		[03:15:33] [03:15:56] 23s
2016-04-30T23:15:57.655 INFO:tasks.rbd.client.0.clara008.stdout:generic/055 220s ...	[03:15:35] [03:15:57] 22s
2016-04-30T23:16:06.586 INFO:tasks.rbd.client.1.clara005.stdout:xfs/295		[03:15:56] [03:16:06] 10s
2016-04-30T23:16:08.032 INFO:tasks.rbd.client.1.clara005.stdout:generic/015		[03:16:06] [03:16:08] 2s
2016-04-30T23:16:08.173 INFO:tasks.rbd.client.0.clara008.stdout:xfs/295 18s ...	[03:15:57] [03:16:08] 11s
2016-04-30T23:16:09.964 INFO:tasks.rbd.client.0.clara008.stdout:generic/015 5s ...	[03:16:08] [03:16:09] 1s
2016-04-30T23:16:12.735 INFO:tasks.rbd.client.1.clara005.stdout:xfs/015		[03:16:08] [03:16:12] 4s
2016-04-30T23:16:15.100 INFO:tasks.rbd.client.0.clara008.stdout:xfs/015		[03:16:10] [03:16:15] 5s
2016-04-30T23:16:17.870 INFO:tasks.rbd.client.1.clara005.stdout:xfs/202		[03:16:16] [03:16:17] 1s
2016-04-30T23:16:18.966 INFO:tasks.rbd.client.1.clara005.stdout:xfs/092		[03:16:17] [03:16:18] 1s
2016-04-30T23:16:19.772 INFO:tasks.rbd.client.0.clara008.stdout:xfs/202 5s ...	[03:16:18] [03:16:19] 1s
2016-04-30T23:16:20.327 INFO:tasks.rbd.client.1.clara005.stdout:generic/141		[03:16:19] [03:16:20] 1s
2016-04-30T23:16:20.563 INFO:tasks.rbd.client.1.clara005.stdout:xfs/042       [expunged]
2016-04-30T23:16:20.571 INFO:tasks.rbd.client.1.clara005.stdout:Ran: xfs/227 xfs/050 generic/127 generic/008 generic/079 generic/010 xfs/293 generic/052 generic/029 generic/319 xfs/060 xfs/188 xfs/121 xfs/196 generic/030 generic/324 generic/286 generic/080 generic/274 generic/120 xfs/290 generic/193 generic/022 xfs/059 xfs/190 generic/037 generic/075 xfs/065 xfs/262 xfs/010 xfs/166 xfs/164 xfs/216 generic/016 xfs/068 xfs/250 generic/020 generic/192 generic/039 xfs/252 xfs/305 generic/041 generic/128 generic/105 xfs/019 generic/032 xfs/054 generic/311 generic/315 xfs/194 generic/249 xfs/199 xfs/266 xfs/004 generic/035 generic/054 xfs/244 generic/071 xfs/031 generic/277 xfs/206 generic/045 generic/077 generic/322 generic/089 generic/074 shared/006 xfs/064 xfs/291 generic/026 generic/014 generic/017 generic/246 generic/018 xfs/187 xfs/002 generic/053 generic/132 generic/133 xfs/084 xfs/304 xfs/282 xfs/183 xfs/013 generic/013 generic/034 generic/076 shared/051 xfs/047 generic/072 generic/308 generic/237 generic/023 generic/131 xfs/009 generic/213 xfs/001 shared/298 generic/223 xfs/134 generic/117 generic/031 generic/260 generic/241 generic/009 generic/285 xfs/049 generic/129 generic/321 generic/215 xfs/201 xfs/296 xfs/008 generic/069 generic/245 xfs/118 generic/256 generic/033 generic/325 generic/263 generic/135 generic/056 generic/236 xfs/046 generic/316 generic/169 generic/002 xfs/292 xfs/027 generic/124 generic/047 generic/306 xfs/063 generic/123 generic/051 xfs/003 xfs/108 generic/003 generic/313 generic/257 xfs/203 generic/294 xfs/229 generic/130 generic/226 xfs/067 generic/248 xfs/110 generic/067 xfs/116 generic/057 xfs/259 xfs/170 xfs/016 xfs/103 generic/059 generic/011 xfs/174 generic/225 generic/066 generic/288 generic/088 generic/025 generic/233 xfs/238 xfs/072 generic/091 xfs/045 generic/021 generic/312 xfs/301 xfs/283 generic/273 generic/320 xfs/056 generic/043 generic/027 xfs/181 xfs/281 shared/032 xfs/078 xfs/062 generic/258 generic/024 generic/065 xfs/253 xfs/041 xfs/014 generic/314 generic/275 xfs/119 xfs/052 xfs/303 generic/001 generic/228 generic/224 generic/005 xfs/278 xfs/299 generic/028 generic/307 xfs/033 generic/232 xfs/005 xfs/186 generic/100 generic/269 xfs/298 xfs/222 xfs/012 generic/270 generic/068 xfs/061 xfs/217 generic/044 xfs/029 generic/070 xfs/034 xfs/006 xfs/048 xfs/066 generic/214 generic/049 generic/234 generic/007 generic/255 generic/062 generic/219 xfs/011 generic/006 xfs/017 xfs/220 generic/221 xfs/302 xfs/020 xfs/026 generic/235 generic/073 generic/040 xfs/021 generic/126 generic/230 generic/309 xfs/165 generic/280 generic/125 generic/012 xfs/028 generic/310 xfs/287 generic/184 generic/055 xfs/295 generic/015 xfs/015 xfs/202 xfs/092 generic/141
2016-04-30T23:16:20.573 INFO:tasks.rbd.client.1.clara005.stdout:Not run: generic/063 generic/318 xfs/189 xfs/040 generic/240 generic/323 generic/212 generic/300 xfs/035 generic/210 generic/211 xfs/195 xfs/044 generic/299 xfs/149 xfs/090 generic/004 xfs/094 generic/207 generic/239 generic/113 xfs/148 xfs/122 generic/058 generic/048 generic/036 generic/208 generic/317 generic/061 generic/112 generic/064 generic/209 generic/198 generic/060
2016-04-30T23:16:20.574 INFO:tasks.rbd.client.1.clara005.stdout:Passed all 258 tests
2016-04-30T23:16:20.575 INFO:tasks.rbd.client.1.clara005.stdout:
2016-04-30T23:16:20.640 INFO:tasks.rbd.client.1.clara005.stdout:/home/ubuntu/cephtest
2016-04-30T23:16:20.649 INFO:tasks.rbd.client.1.clara005.stdout:This xfstests run started at:  Sun May  1 01:24:42 UTC 2016
2016-04-30T23:16:20.667 INFO:tasks.rbd.client.1.clara005.stdout:xfstests run completed at:     Sun May  1 03:16:20 UTC 2016
2016-04-30T23:16:20.708 INFO:tasks.rbd.client.1.clara005.stderr:device "/dev/rbd0" was not mounted
2016-04-30T23:16:20.709 INFO:tasks.rbd.client.1.clara005.stderr:device "/dev/rbd1" was not mounted
2016-04-30T23:16:20.752 INFO:tasks.rbd.client.0.clara008.stdout:xfs/092 3s ...	[03:16:19] [03:16:20] 1s
2016-04-30T23:16:20.794 INFO:tasks.rbd.client.1.clara005.stderr:+ STATUS=0
2016-04-30T23:16:20.795 INFO:tasks.rbd.client.1.clara005.stderr:+ rm -f expunge.W2NunG5fq8
2016-04-30T23:16:20.797 INFO:tasks.rbd.client.1.clara005.stderr:+ rm -f run_xfstests.sh
2016-04-30T23:16:20.798 INFO:tasks.rbd.client.1.clara005.stderr:+ exit 0
2016-04-30T23:16:20.800 INFO:tasks.rbd:Removing run_xfstests_krbd.sh on client.1
2016-04-30T23:16:20.802 INFO:teuthology.orchestra.run.clara005:Running: 'rm -f /home/ubuntu/cephtest/run_xfstests_krbd.sh'
2016-04-30T23:16:21.909 INFO:tasks.rbd.client.0.clara008.stdout:generic/141 3s ...	[03:16:20] [03:16:21] 1s
2016-04-30T23:16:22.123 INFO:tasks.rbd.client.0.clara008.stdout:xfs/042       [expunged]
2016-04-30T23:16:22.132 INFO:tasks.rbd.client.0.clara008.stdout:Ran: xfs/227 xfs/050 generic/127 generic/008 generic/079 generic/010 xfs/293 generic/052 generic/029 generic/319 xfs/060 xfs/188 xfs/121 xfs/196 generic/030 generic/324 generic/286 generic/080 generic/274 generic/120 xfs/290 generic/193 generic/022 xfs/059 xfs/190 generic/037 generic/075 xfs/065 xfs/262 xfs/010 xfs/166 xfs/164 xfs/216 generic/016 xfs/068 xfs/250 generic/020 generic/192 generic/039 xfs/252 xfs/305 generic/041 generic/128 generic/105 xfs/019 generic/032 xfs/054 generic/311 generic/315 xfs/194 generic/249 xfs/199 xfs/266 xfs/004 generic/035 generic/054 xfs/244 generic/071 xfs/031 generic/277 xfs/206 generic/045 generic/077 generic/322 generic/089 generic/074 shared/006 xfs/064 xfs/291 generic/026 generic/014 generic/017 generic/246 generic/018 xfs/187 xfs/002 generic/053 generic/132 generic/133 xfs/084 xfs/304 xfs/282 xfs/183 xfs/013 generic/013 generic/034 generic/076 shared/051 xfs/047 generic/072 generic/308 generic/237 generic/023 generic/131 xfs/009 generic/213 xfs/001 shared/298 generic/223 xfs/134 generic/117 generic/031 generic/260 generic/241 generic/009 generic/285 xfs/049 generic/129 generic/321 generic/215 xfs/201 xfs/296 xfs/008 generic/069 generic/245 xfs/118 generic/256 generic/033 generic/325 generic/263 generic/135 generic/056 generic/236 xfs/046 generic/316 generic/169 generic/002 xfs/292 xfs/027 generic/124 generic/047 generic/306 xfs/063 generic/123 generic/051 xfs/003 xfs/108 generic/003 generic/313 generic/257 xfs/203 generic/294 xfs/229 generic/130 generic/226 xfs/067 generic/248 xfs/110 generic/067 xfs/116 generic/057 xfs/259 xfs/170 xfs/016 xfs/103 generic/059 generic/011 xfs/174 generic/225 generic/066 generic/288 generic/088 generic/025 generic/233 xfs/238 xfs/072 generic/091 xfs/045 generic/021 generic/312 xfs/301 xfs/283 generic/273 generic/320 xfs/056 generic/043 generic/027 xfs/181 xfs/281 shared/032 xfs/078 xfs/062 generic/258 generic/024 generic/065 xfs/253 xfs/041 xfs/014 generic/314 generic/275 xfs/119 xfs/052 xfs/303 generic/001 generic/228 generic/224 generic/005 xfs/278 xfs/299 generic/028 generic/307 xfs/033 generic/232 xfs/005 xfs/186 generic/100 generic/269 xfs/298 xfs/222 xfs/012 generic/270 generic/068 xfs/061 xfs/217 generic/044 xfs/029 generic/070 xfs/034 xfs/006 xfs/048 xfs/066 generic/214 generic/049 generic/234 generic/007 generic/255 generic/062 generic/219 xfs/011 generic/006 xfs/017 xfs/220 generic/221 xfs/302 xfs/020 xfs/026 generic/235 generic/073 generic/040 xfs/021 generic/126 generic/230 generic/309 xfs/165 generic/280 generic/125 generic/012 xfs/028 generic/310 xfs/287 generic/184 generic/055 xfs/295 generic/015 xfs/015 xfs/202 xfs/092 generic/141
2016-04-30T23:16:22.133 INFO:tasks.rbd.client.0.clara008.stdout:Not run: generic/063 generic/318 xfs/189 xfs/040 generic/240 generic/323 generic/212 generic/300 xfs/035 generic/210 generic/211 xfs/195 xfs/044 generic/299 xfs/149 xfs/090 generic/004 xfs/094 generic/207 generic/239 generic/113 xfs/148 xfs/122 generic/058 generic/048 generic/036 generic/208 generic/317 generic/061 generic/112 generic/064 generic/209 generic/198 generic/060
2016-04-30T23:16:22.134 INFO:tasks.rbd.client.0.clara008.stdout:Failures: generic/312 generic/310
2016-04-30T23:16:22.136 INFO:tasks.rbd.client.0.clara008.stdout:Failed 2 of 258 tests

Version-Release number of selected component (if applicable):
ceph version 10.2.0-1.el7cp (3a9fba20ec743699b69bd0181dd6c54dc01c64b9)


How reproducible:
1/1

Steps to Reproduce:
 krbd/singleton/{conf.yaml fs/btrfs.yaml msgr-failures/few.yaml tasks/rbd_xfstests.yaml}


Additional info:
http://magna002.ceph.redhat.com/vasu-2016-04-30_14:48:23-krbd-jewel---basic-clara/229816/teuthology.log
Comment 2 Brad Hubbard 2016-05-03 22:06:54 UTC 
Hi Vasu,

These appear to be the relevant sections of the log.

2016-04-30T22:44:58.767 INFO:tasks.rbd.client.0.clara008.stdout:generic/312 2s ...	[02:44:54] [02:44:58] 4s
2016-04-30T22:44:58.819 INFO:tasks.rbd.client.0.clara008.stdout:_check_xfs_filesystem: filesystem on /dev/rbd2 is inconsistent (c) (see /var/lib/xfstests/results//generic/312.full)

2016-04-30T23:15:32.241 INFO:tasks.rbd.client.0.clara008.stdout:generic/310 64s ...	[03:14:57] [03:15:32] [failed, exit status 1] - output mismatch (see /var/lib/xfstests/results//generic/310.out.bad)
2016-04-30T23:15:32.260 INFO:tasks.rbd.client.0.clara008.stdout:    --- tests/generic/310.out	2016-05-01 01:27:12.725451314 +0000
2016-04-30T23:15:32.288 INFO:tasks.rbd.client.0.clara008.stdout:    +++ /var/lib/xfstests/results//generic/310.out.bad	2016-05-01 03:15:32.237191123 +0000
2016-04-30T23:15:32.329 INFO:tasks.rbd.client.0.clara008.stdout:    @@ -1,2 +1,2 @@
2016-04-30T23:15:32.329 INFO:tasks.rbd.client.0.clara008.stdout:     QA output created by 310
2016-04-30T23:15:32.330 INFO:tasks.rbd.client.0.clara008.stdout:    -*** done
2016-04-30T23:15:32.331 INFO:tasks.rbd.client.0.clara008.stdout:    +kernel bug detected, check dmesg for more infomation.
2016-04-30T23:15:32.332 INFO:tasks.rbd.client.0.clara008.stdout:    ...
2016-04-30T23:15:32.333 INFO:tasks.rbd.client.0.clara008.stdout:    (Run 'diff -u tests/generic/310.out /var/lib/xfstests/results//generic/310.out.bad'  to see the entire diff)

Could you post the contents of /var/lib/xfstests/results//generic/312.full and 'diff -u tests/generic/310.out /var/lib/xfstests/results//generic/310.out.bad'?
 
310 looks like it is reporting a kernel bug.
Comment 3 Vasu Kulkarni 2016-05-04 21:50:20 UTC 
Brad,

Thats a good point, It didnt collect those log files, I am going to run this again and get the diff.
Note You need to log in before you can comment on or make changes to this bug.