Bug 1319266

Summary: [RH Ceph 2.0] rbd fsx - ceph_test_librbd_fsx failed
Product: Red Hat Ceph Storage Reporter: Vasu Kulkarni <vakulkar>
Component: RBDAssignee: Josh Durgin <jdurgin>
Status: CLOSED CURRENTRELEASE QA Contact: ceph-qe-bugs <ceph-qe-bugs>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 2.0CC: ceph-eng-bugs, hnallurv, icolle, jdillama, nlevine, vakulkar
Target Milestone: rc   
Target Release: 2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-04-11 18:48:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Vasu Kulkarni 2016-03-18 16:24:56 UTC 
Description of problem:

rbd fsx - ceph_tst_librbd_fsx failed

2016-03-18T03:11:02.850 INFO:teuthology.orchestra.run.clara010.stdout:READ BAD DATA: offset = 0x0, size = 0xde78d92, fname = image_client.0
2016-03-18T03:11:02.851 INFO:teuthology.orchestra.run.clara010.stdout:OFFSET	GOOD	BAD	RANGE
2016-03-18T03:11:02.858 INFO:teuthology.orchestra.run.clara010.stdout:0x52dbd1	0x0000	0x0e39	0x    0
2016-03-18T03:11:02.859 INFO:teuthology.orchestra.run.clara010.stdout:operation# (mod 256) for the bad data may be 57
2016-03-18T03:11:02.860 INFO:teuthology.orchestra.run.clara010.stdout:0x52dbd2	0x0000	0x39ea	0x    1
2016-03-18T03:11:02.861 INFO:teuthology.orchestra.run.clara010.stdout:operation# (mod 256) for the bad data may be 57
2016-03-18T03:11:02.862 INFO:teuthology.orchestra.run.clara010.stdout:0x52dbd3	0x0000	0xea39	0x    2
2016-03-18T03:11:02.863 INFO:teuthology.orchestra.run.clara010.stdout:operation# (mod 256) for the bad data may be 57
2016-03-18T03:11:02.864 INFO:teuthology.orchestra.run.clara010.stdout:0x52dbd4	0x0000	0x3927	0x    3
2016-03-18T03:11:02.865 INFO:teuthology.orchestra.run.clara010.stdout:operation# (mod 256) for the bad data may be 57
2016-03-18T03:11:02.866 INFO:teuthology.orchestra.run.clara010.stdout:0x52dbd5	0x0000	0x2739	0x    4
2016-03-18T03:11:02.867 INFO:teuthology.orchestra.run.clara010.stdout:operation# (mod 256) for the bad data may be 57
2016-03-18T03:11:02.868 INFO:teuthology.orchestra.run.clara010.stdout:0x52dbd6	0x0000	0x3984	0x    5
2016-03-18T03:11:02.869 INFO:teuthology.orchestra.run.clara010.stdout:operation# (mod 256) for the bad data may be 57
2016-03-18T03:11:02.869 INFO:teuthology.orchestra.run.clara010.stdout:0x52dbd7	0x0000	0x8439	0x    6
2016-03-18T03:11:02.871 INFO:teuthology.orchestra.run.clara010.stdout:operation# (mod 256) for the bad data may be 57
2016-03-18T03:11:02.871 INFO:teuthology.orchestra.run.clara010.stdout:0x52dbd8	0x0000	0x39b8	0x    7
2016-03-18T03:11:02.872 INFO:teuthology.orchestra.run.clara010.stdout:operation# (mod 256) for the bad data may be 57
2016-03-18T03:11:02.873 INFO:teuthology.orchestra.run.clara010.stdout:0x52dbd9	0x0000	0xb839	0x    8
2016-03-18T03:11:02.874 INFO:teuthology.orchestra.run.clara010.stdout:operation# (mod 256) for the bad data may be 57
2016-03-18T03:11:02.875 INFO:teuthology.orchestra.run.clara010.stdout:0x52dbda	0x0000	0x3936	0x    9
2016-03-18T03:11:02.876 INFO:teuthology.orchestra.run.clara010.stdout:operation# (mod 256) for the bad data may be 57
2016-03-18T03:11:02.877 INFO:teuthology.orchestra.run.clara010.stdout:0x52dbdb	0x0000	0x3639	0x    a
2016-03-18T03:11:02.878 INFO:teuthology.orchestra.run.clara010.stdout:operation# (mod 256) for the bad data may be 57
2016-03-18T03:11:02.879 INFO:teuthology.orchestra.run.clara010.stdout:0x52dbdc	0x0000	0x3995	0x    b
2016-03-18T03:11:02.879 INFO:teuthology.orchestra.run.clara010.stdout:operation# (mod 256) for the bad data may be 57
2016-03-18T03:11:02.880 INFO:teuthology.orchestra.run.clara010.stdout:0x52dbdd	0x0000	0x9539	0x    c
2016-03-18T03:11:02.881 INFO:teuthology.orchestra.run.clara010.stdout:operation# (mod 256) for the bad data may be 57
2016-03-18T03:11:02.882 INFO:teuthology.orchestra.run.clara010.stdout:0x52dbde	0x0000	0x39cd	0x    d
2016-03-18T03:11:02.883 INFO:teuthology.orchestra.run.clara010.stdout:operation# (mod 256) for the bad data may be 57
2016-03-18T03:11:02.884 INFO:teuthology.orchestra.run.clara010.stdout:0x52dbdf	0x0000	0xcd39	0x    e
2016-03-18T03:11:02.885 INFO:teuthology.orchestra.run.clara010.stdout:operation# (mod 256) for the bad data may be 57
2016-03-18T03:11:02.886 INFO:teuthology.orchestra.run.clara010.stdout:0x52dbe0	0x0000	0x3985	0x    f
2016-03-18T03:11:02.887 INFO:teuthology.orchestra.run.clara010.stdout:operation# (mod 256) for the bad data may be 57
2016-03-18T03:11:03.175 INFO:teuthology.orchestra.run.clara010.stdout:LOG DUMP (166 total operations):
2016-03-18T03:11:03.176 INFO:teuthology.orchestra.run.clara010.stdout:1(  1 mod 256): SKIPPED (no operation)
2016-03-18T03:11:03.177 INFO:teuthology.orchestra.run.clara010.stdout:2(  2 mod 256): SKIPPED (no operation)

2016-03-18T03:11:03.296 INFO:teuthology.orchestra.run.clara010.stdout:143(143 mod 256): TRUNCATE DOWN	from 0xb9327a5 to 0x8c235ac	******WWWW
2016-03-18T03:11:03.297 INFO:teuthology.orchestra.run.clara010.stdout:144(144 mod 256): SKIPPED (no operation)
2016-03-18T03:11:03.298 INFO:teuthology.orchestra.run.clara010.stdout:145(145 mod 256): PUNCH    0x1f0c601 thru 0x1f1633a	(0x9d3a bytes)
2016-03-18T03:11:03.299 INFO:teuthology.orchestra.run.clara010.stdout:146(146 mod 256): SKIPPED (no operation)
2016-03-18T03:11:03.300 INFO:teuthology.orchestra.run.clara010.stdout:147(147 mod 256): WRITE    0xd52d0dd thru 0xd53104e	(0x3f72 bytes) HOLE	***WWWW
2016-03-18T03:11:03.300 INFO:teuthology.orchestra.run.clara010.stdout:148(148 mod 256): WRITE    0xfaab4f thru 0xfb42c2	(0x9774 bytes)
2016-03-18T03:11:03.301 INFO:teuthology.orchestra.run.clara010.stdout:149(149 mod 256): READ     0xbc2c6a8 thru 0xbc3485f	(0x81b8 bytes)
2016-03-18T03:11:03.302 INFO:teuthology.orchestra.run.clara010.stdout:150(150 mod 256): PUNCH    0x54487d8 thru 0x544ac13	(0x243c bytes)
2016-03-18T03:11:03.303 INFO:teuthology.orchestra.run.clara010.stdout:151(151 mod 256): SKIPPED (no operation)
2016-03-18T03:11:03.303 INFO:teuthology.orchestra.run.clara010.stdout:152(152 mod 256): SKIPPED (no operation)
2016-03-18T03:11:03.304 INFO:teuthology.orchestra.run.clara010.stdout:153(153 mod 256): TRUNCATE DOWN	from 0xd53104f to 0x8a06a83	******WWWW
2016-03-18T03:11:03.305 INFO:teuthology.orchestra.run.clara010.stdout:154(154 mod 256): SKIPPED (no operation)
2016-03-18T03:11:03.306 INFO:teuthology.orchestra.run.clara010.stdout:155(155 mod 256): SKIPPED (no operation)
2016-03-18T03:11:03.306 INFO:teuthology.orchestra.run.clara010.stdout:156(156 mod 256): WRITE    0x946f9de thru 0x947b324	(0xb947 bytes) HOLE	***WWWW
2016-03-18T03:11:03.307 INFO:teuthology.orchestra.run.clara010.stdout:157(157 mod 256): SKIPPED (no operation)
2016-03-18T03:11:03.308 INFO:teuthology.orchestra.run.clara010.stdout:158(158 mod 256): TRUNCATE UP	from 0x947b325 to 0x94c4c0c
2016-03-18T03:11:03.309 INFO:teuthology.orchestra.run.clara010.stdout:159(159 mod 256): READ     0x3874d47 thru 0x3877c61	(0x2f1b bytes)
2016-03-18T03:11:03.310 INFO:teuthology.orchestra.run.clara010.stdout:160(160 mod 256): TRUNCATE DOWN	from 0x94c4c0c to 0x574620d	******WWWW
2016-03-18T03:11:03.311 INFO:teuthology.orchestra.run.clara010.stdout:161(161 mod 256): SKIPPED (no operation)
2016-03-18T03:11:03.312 INFO:teuthology.orchestra.run.clara010.stdout:162(162 mod 256): READ     0xe846df thru 0xe90d4a	(0xc66c bytes)
2016-03-18T03:11:03.312 INFO:teuthology.orchestra.run.clara010.stdout:163(163 mod 256): WRITE    0xc030f52 thru 0xc03feb8	(0xef67 bytes) HOLE	***WWWW
2016-03-18T03:11:03.313 INFO:teuthology.orchestra.run.clara010.stdout:164(164 mod 256): READ     0x362d1b7 thru 0x3633933	(0x677d bytes)
2016-03-18T03:11:03.314 INFO:teuthology.orchestra.run.clara010.stdout:165(165 mod 256): SKIPPED (no operation)
2016-03-18T03:11:03.314 INFO:teuthology.orchestra.run.clara010.stdout:166(166 mod 256): CLONE
2016-03-18T03:11:03.317 INFO:teuthology.orchestra.run.clara011.stderr:set osd.0 primary-affinity to 0 (802)
2016-03-18T03:11:03.341 INFO:teuthology.orchestra.run.clara010.stdout:Correct content saved for comparison
2016-03-18T03:11:03.342 INFO:teuthology.orchestra.run.clara010.stdout:(maybe hexdump "image_client.0" vs "image_client.0.fsxgood")
2016-03-18T03:11:06.350 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_wip-ver-update/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/teuthworker/src/teuthology_wip-ver-update/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/teuthology_wip-ver-update/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/ceph-qa-suite_jewel/tasks/rbd_fsx.py", line 99, in _run_one_client
    remote.run(args=args)
  File "/home/teuthworker/src/teuthology_wip-ver-update/teuthology/orchestra/remote.py", line 196, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_wip-ver-update/teuthology/orchestra/run.py", line 378, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_wip-ver-update/teuthology/orchestra/run.py", line 114, in wait
    label=self.label)
CommandFailedError: Command failed on clara010 with status 110: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_librbd_fsx -d -W -R -p 100 -P /home/ubuntu/cephtest/archive -r 1 -w 1 -t 1 -h 1 -l 250000000 -S 0 -N 2000 pool_client.0 image_client.0'
2016-03-18T03:11:

Version-Release number of selected component (if applicable):
10.0.4

How reproducible:
Unknown

Steps to Reproduce:
Running the rbd_fsx test

Actual results:
CommandFailedError: Command failed on clara010 with status 110: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_librbd_fsx -d -W -R -p 100 -P /home/ubuntu/cephtest/archive -r 1 -w 1 -t 1 -h 1 -l 250000000 -S 0 -N 2000 pool_client.0 image_client.0'

Expected results:


Additional info:
http://magna002.ceph.redhat.com/vasu-2016-03-17_19:50:58-smoke-master---basic-clara/215660/teuthology.log
Comment 2 Jason Dillaman 2016-04-11 17:29:32 UTC 
Is this still an issue on the 10.1.x release?
Comment 3 Vasu Kulkarni 2016-04-11 18:48:39 UTC 
This works in 10.1.1 rebase.