| Summary: | [RH Ceph 2.0 - 10.1.0] LibRadosTwoPoolsPP.Whiteout and LibRadosTwoPoolsECPP.Whiteout Failed | ||
|---|---|---|---|
| Product: | Red Hat Ceph Storage | Reporter: | Vasu Kulkarni <vakulkar> |
| Component: | RADOS | Assignee: | Samuel Just <sjust> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | ceph-qe-bugs <ceph-qe-bugs> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 2.0 | CC: | ceph-eng-bugs, dzafman, kchai, kdreyer |
| Target Milestone: | rc | ||
| Target Release: | 2.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-10-05 15:50:07 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
Test bug, should have been fixed in c335855e971334c76abd13c722d57ee9e423ea74 (not in the 10.1.0 branch you are testing) http://tracker.ceph.com/issues/15177 Verified in 10.1.1 Fixed in the RHCS 2 GA. https://access.redhat.com/errata/RHBA-2016:1755 |
Description of problem: During Rados smoke test following tests failed 2016-04-04T16:43:07.811 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ PASSED ] 44 tests. 2016-04-04T16:43:07.812 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ FAILED ] 2 tests, listed below: 2016-04-04T16:43:07.814 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ FAILED ] LibRadosTwoPoolsPP.Whiteout 2016-04-04T16:43:07.815 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ FAILED ] LibRadosTwoPoolsECPP.Whiteout Version-Release number of selected component (if applicable): ceph version 10.1.0-1.el7cp (61adb020219fbad4508050b5f0a792246ba74dae) How reproducible: 10.1.0 Steps to Reproduce: Run Rados smoke test Actual results: 2016-04-04T16:41:56.853 INFO:teuthology.orchestra.run.clara009:Running: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' 2016-04-04T16:41:56.860 INFO:teuthology.orchestra.run.clara010:Running: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' 2016-04-04T16:42:14.252 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValuePP (7034 ms) 2016-04-04T16:42:14.254 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2016-04-04T16:42:14.255 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7028 ms) 2016-04-04T16:42:14.256 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushPP 2016-04-04T16:42:14.257 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ OK ] LibRadosAioEC.FlushPP (7119 ms) 2016-04-04T16:42:14.258 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2016-04-04T16:42:14.259 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (8548 ms) 2016-04-04T16:42:14.259 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsyncPP 2016-04-04T16:42:14.260 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsyncPP (7043 ms) 2016-04-04T16:42:14.261 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2016-04-04T16:42:14.262 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7032 ms) 2016-04-04T16:42:14.263 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2016-04-04T16:42:14.264 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7044 ms) 2016-04-04T16:42:14.265 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2016-04-04T16:42:14.265 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3034 ms) 2016-04-04T16:42:14.266 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2016-04-04T16:42:14.267 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7178 ms) 2016-04-04T16:42:14.268 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatPP 2016-04-04T16:42:14.269 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatPP (7851 ms) 2016-04-04T16:42:14.270 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2016-04-04T16:42:14.270 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (35233 ms) 2016-04-04T16:42:14.271 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2016-04-04T16:42:14.272 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatPPNS (23161 ms) 2016-04-04T16:42:14.273 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2016-04-04T16:42:14.274 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7294 ms) 2016-04-04T16:42:14.274 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemovePP 2016-04-04T16:42:14.275 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemovePP (7257 ms) 2016-04-04T16:42:14.276 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ RUN ] LibRadosAioEC.OmapPP 2016-04-04T16:42:14.277 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ OK ] LibRadosAioEC.OmapPP (5035 ms) 2016-04-04T16:42:14.278 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2016-04-04T16:42:14.279 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7048 ms) 2016-04-04T16:42:14.279 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWritePP 2016-04-04T16:42:14.280 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWritePP (7155 ms) 2016-04-04T16:42:14.281 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [----------] 34 tests from LibRadosAioEC (304738 ms total) 2016-04-04T16:42:14.282 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: 2016-04-04T16:42:14.283 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [----------] Global test environment tear-down 2016-04-04T16:42:14.284 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [==========] 71 tests from 2 test cases ran. (598830 ms total) 2016-04-04T16:42:14.285 INFO:tasks.workunit.client.0.clara010.stdout: api_aio: [ PASSED ] 71 tests. 2016-04-04T16:42:14.285 INFO:tasks.workunit.client.0.clara010.stderr:+ for p in '$pids' 2016-04-04T16:42:14.286 INFO:tasks.workunit.client.0.clara010.stderr:+ wait 23142 2016-04-04T16:42:14.287 INFO:tasks.workunit.client.0.clara010.stderr:+ for p in '$pids' 2016-04-04T16:42:14.288 INFO:tasks.workunit.client.0.clara010.stderr:+ wait 23144 2016-04-04T16:42:14.289 INFO:tasks.workunit.client.0.clara010.stderr:+ for p in '$pids' 2016-04-04T16:42:14.290 INFO:tasks.workunit.client.0.clara010.stderr:+ wait 23155 2016-04-04T16:42:14.291 INFO:tasks.workunit.client.0.clara010.stderr:+ for p in '$pids' 2016-04-04T16:42:14.292 INFO:tasks.workunit.client.0.clara010.stderr:+ wait 23160 2016-04-04T16:42:14.292 INFO:tasks.workunit.client.0.clara010.stderr:+ for p in '$pids' 2016-04-04T16:42:14.293 INFO:tasks.workunit.client.0.clara010.stderr:+ wait 23165 2016-04-04T16:42:19.322 INFO:tasks.ceph.osd.1.clara002.stderr:2016-04-04 20:42:19.320330 7fe360bcd700 -1 osd.1 484 promote_throttle_recalibrate po 948757463 pb 3125 avg_size 8 2016-04-04T16:42:22.284 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ RUN ] LibRadosTwoPoolsECPP.Promote 2016-04-04T16:42:22.285 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ OK ] LibRadosTwoPoolsECPP.Promote (22141 ms) 2016-04-04T16:42:22.286 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2016-04-04T16:42:22.287 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: waiting for scrub... 2016-04-04T16:42:22.288 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: done waiting 2016-04-04T16:42:22.289 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (24136 ms) 2016-04-04T16:42:22.290 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2016-04-04T16:42:22.291 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (8997 ms) 2016-04-04T16:42:22.292 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2016-04-04T16:42:22.293 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: test/librados/tier.cc:3192: Failure 2016-04-04T16:42:22.294 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: Value of: it != cache_ioctx.nobjects_end() 2016-04-04T16:42:22.295 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: Actual: false 2016-04-04T16:42:22.296 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: Expected: true 2016-04-04T16:42:22.298 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ FAILED ] LibRadosTwoPoolsECPP.Whiteout (7024 ms) 2016-04-04T16:42:22.299 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ RUN ] LibRadosTwoPoolsECPP.Evict 2016-04-04T16:42:22.300 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ OK ] LibRadosTwoPoolsECPP.Evict (7117 ms) 2016-04-04T16:42:22.301 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2016-04-04T16:42:22.302 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (9495 ms) 2016-04-04T16:42:22.303 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2016-04-04T16:42:22.304 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ OK ] LibRadosTwoPoolsECPP.TryFlush (7106 ms) 2016-04-04T16:42:22.305 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2016-04-04T16:42:22.306 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (12060 ms) 2016-04-04T16:42:22.307 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ RUN ] LibRadosTwoPoolsECPP.Flush 2016-04-04T16:42:22.308 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ OK ] LibRadosTwoPoolsECPP.Flush (7065 ms) 2016-04-04T16:42:22.309 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2016-04-04T16:42:22.310 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (13018 ms) 2016-04-04T16:42:22.311 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2016-04-04T16:42:22.312 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (32204 ms) 2016-04-04T16:42:22.313 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2016-04-04T16:42:22.314 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (28118 ms) 2016-04-04T16:42:22.316 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2016-04-04T16:42:22.317 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: hmm, no HitSet yet 2016-04-04T16:42:22.318 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: ok, hit_set contains 180:602f83fe:::foo:head 2016-04-04T16:42:22.319 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (8385 ms) 2016-04-04T16:42:22.320 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2016-04-04T16:42:22.321 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: got ls 1459802516,0 2016-04-04T16:42:22.321 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: first is 1459802516 2016-04-04T16:42:22.322 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: got ls 1459802516,0 2016-04-04T16:42:22.323 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: got ls 1459802516,0 2016-04-04T16:42:22.324 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: got ls 1459802516,1459802518,1459802519,0 2016-04-04T16:42:22.325 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: got ls 1459802516,1459802518,1459802519,0 2016-04-04T16:42:22.326 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: got ls 1459802516,1459802518,1459802519,0 2016-04-04T16:42:22.327 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: got ls 1459802516,1459802518,1459802519,1459802521,1459802522,0 2016-04-04T16:42:22.328 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: got ls 1459802516,1459802518,1459802519,1459802521,1459802522,0 2016-04-04T16:42:22.329 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: got ls 1459802516,1459802518,1459802519,1459802521,1459802522,0 2016-04-04T16:42:22.329 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: got ls 1459802516,1459802518,1459802519,1459802521,1459802522,1459802524,1459802525,0 2016-04-04T16:42:22.330 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: got ls 1459802516,1459802518,1459802519,1459802521,1459802522,1459802524,1459802525,0 2016-04-04T16:42:22.331 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: got ls 1459802516,1459802518,1459802519,1459802521,1459802522,1459802524,1459802525,0 2016-04-04T16:42:22.332 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: got ls 1459802519,1459802521,1459802522,1459802524,1459802525,1459802527,1459802528,0 2016-04-04T16:42:22.333 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: first now 1459802519, trimmed 2016-04-04T16:42:22.334 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (20375 ms) 2016-04-04T16:42:22.335 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2016-04-04T16:42:22.336 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: foo0 2016-04-04T16:42:22.337 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: verifying foo0 is eventually promoted 2016-04-04T16:42:22.338 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (13141 ms) 2016-04-04T16:42:26.932 INFO:teuthology.orchestra.run.clara002:Running: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' 2016-04-04T16:42:26.937 INFO:teuthology.orchestra.run.clara009:Running: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' 2016-04-04T16:42:26.942 INFO:teuthology.orchestra.run.clara010:Running: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' 2016-04-04T16:42:45.622 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2016-04-04T16:42:45.623 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2016-04-04T16:42:45.624 INFO:tasks.ceph.mon.c:Restarting daemon 2016-04-04T16:42:45.625 INFO:teuthology.orchestra.run.clara009:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f -i c' 2016-04-04T16:42:45.630 INFO:tasks.ceph.mon.c:Started 2016-04-04T16:42:45.631 INFO:tasks.mon_thrash:Sending STOP to mon b 2016-04-04T16:42:45.632 INFO:tasks.ceph.mon.b:Sent signal 19 2016-04-04T16:42:45.633 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2016-04-04T16:42:45.803 INFO:tasks.ceph.mon.c.clara009.stdout:starting mon.c rank 2 at 10.8.129.9:6790/0 mon_data /var/lib/ceph/mon/ceph-c fsid 532ae0c0-b1c3-4aec-b31e-b28b47ea8135 2016-04-04T16:42:57.013 INFO:teuthology.orchestra.run.clara002:Running: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' 2016-04-04T16:42:57.020 INFO:teuthology.orchestra.run.clara009:Running: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' 2016-04-04T16:42:57.027 INFO:teuthology.orchestra.run.clara010:Running: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' 2016-04-04T16:43:00.635 INFO:tasks.mon_thrash:Sending CONT to mon b 2016-04-04T16:43:00.637 INFO:tasks.ceph.mon.b:Sent signal 18 2016-04-04T16:43:00.638 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2016-04-04T16:43:00.639 INFO:teuthology.orchestra.run.clara002:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph quorum_status' 2016-04-04T16:43:01.873 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","monmap":{"epoch":1,"fsid":"532ae0c0-b1c3-4aec-b31e-b28b47ea8135","modified":"2016-04-04 20:31:26.225792","created":"2016-04-04 20:31:26.225792","mons":[{"rank":0,"name":"a","addr":"10.8.129.2:6789\/0"},{"rank":1,"name":"b","addr":"10.8.129.9:6789\/0"},{"rank":2,"name":"c","addr":"10.8.129.9:6790\/0"}]}} 2016-04-04T16:43:01.874 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2016-04-04T16:43:01.875 INFO:teuthology.orchestra.run.clara002:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph -m 10.8.129.2:6789 mon_status' 2016-04-04T16:43:03.104 INFO:teuthology.orchestra.run.clara002:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph -m 10.8.129.9:6789 mon_status' 2016-04-04T16:43:04.335 INFO:teuthology.orchestra.run.clara002:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph -m 10.8.129.9:6790 mon_status' 2016-04-04T16:43:05.567 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2016-04-04T16:43:05.568 INFO:teuthology.orchestra.run.clara002:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph scrub' 2016-04-04T16:43:06.800 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2016-04-04T16:43:07.802 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2016-04-04T16:43:07.803 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (16168 ms) 2016-04-04T16:43:07.804 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2016-04-04T16:43:07.805 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ OK ] LibRadosTwoPoolsECPP.CachePin (26315 ms) 2016-04-04T16:43:07.807 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [----------] 18 tests from LibRadosTwoPoolsECPP (270236 ms total) 2016-04-04T16:43:07.808 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: 2016-04-04T16:43:07.809 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [----------] Global test environment tear-down 2016-04-04T16:43:07.810 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [==========] 46 tests from 4 test cases ran. (652312 ms total) 2016-04-04T16:43:07.811 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ PASSED ] 44 tests. 2016-04-04T16:43:07.812 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ FAILED ] 2 tests, listed below: 2016-04-04T16:43:07.814 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ FAILED ] LibRadosTwoPoolsPP.Whiteout 2016-04-04T16:43:07.815 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: [ FAILED ] LibRadosTwoPoolsECPP.Whiteout 2016-04-04T16:43:07.816 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: 2016-04-04T16:43:07.817 INFO:tasks.workunit.client.0.clara010.stdout: api_tier: 2 FAILED TESTS 2016-04-04T16:43:07.818 INFO:tasks.workunit.client.0.clara010.stdout:error in 23165 2016-04-04T16:43:07.819 INFO:tasks.workunit.client.0.clara010.stderr:+ echo 'error in 23165' 2016-04-04T16:43:07.820 INFO:tasks.workunit.client.0.clara010.stderr:+ ret=1 2016-04-04T16:43:07.821 INFO:tasks.workunit.client.0.clara010.stderr:+ for p in '$pids' 2016-04-04T16:43:07.822 INFO:tasks.workunit.client.0.clara010.stderr:+ wait 23174 2016-04-04T16:43:07.822 INFO:tasks.workunit.client.0.clara010.stderr:+ for p in '$pids' 2016-04-04T16:43:07.823 INFO:tasks.workunit.client.0.clara010.stderr:+ wait 23176 2016-04-04T16:43:07.824 INFO:tasks.workunit.client.0.clara010.stderr:+ for p in '$pids' 2016-04-04T16:43:07.825 INFO:tasks.workunit.client.0.clara010.stderr:+ wait 23183 2016-04-04T16:43:07.826 INFO:tasks.workunit.client.0.clara010.stderr:+ for p in '$pids' 2016-04-04T16:43:07.827 INFO:tasks.workunit.client.0.clara010.stderr:+ wait 23209 2016-04-04T16:43:07.828 INFO:tasks.workunit.client.0.clara010.stderr:+ for p in '$pids' 2016-04-04T16:43:07.829 INFO:tasks.workunit.client.0.clara010.stderr:+ wait 23220 2016-04-04T16:43:07.830 INFO:tasks.workunit.client.0.clara010.stderr:+ for p in '$pids' 2016-04-04T16:43:07.831 INFO:tasks.workunit.client.0.clara010.stderr:+ wait 23225 2016-04-04T16:43:07.832 INFO:tasks.workunit.client.0.clara010.stderr:+ for p in '$pids' Expected results: Additional info: http://magna002.ceph.redhat.com/vasu-2016-04-04_15:32:20-smoke-jewel---basic-clara/217127/teuthology.log