Bug 1354616

Summary: [RH Ceph 2.0 / 10.2.2-15] TestLibRBD.SnapCreateViaLockOwner failed in writeback cache
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Vasu Kulkarni <vakulkar>
Component: RBDAssignee: Jason Dillaman <jdillama>
Status: CLOSED ERRATA QA Contact: Vasu Kulkarni <vakulkar>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 2.0CC: ceph-eng-bugs, hnallurv, kdreyer, vakulkar
Target Milestone: rc   
Target Release: 2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: RHEL: ceph-10.2.2-19.el7cp Ubuntu: ceph_10.2.2-15redhat1xenial Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-08-23 19:44:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Vasu Kulkarni 2016-07-11 17:52:13 UTC
Description of problem:
During full rbd regression run on 10.2.2-15, librbd test failed for following test(writeback cache)

2016-07-10T18:10:50.250 INFO:tasks.workunit.client.0.clara005.stdout:[  FAILED  ] 1 test, listed below:
2016-07-10T18:10:50.251 INFO:tasks.workunit.client.0.clara005.stdout:[  FAILED  ] TestLibRBD.SnapCreateViaLockOwner

======

2016-07-10T18:10:40.213 INFO:tasks.workunit.client.0.clara005.stdout:[       OK ] TestJournalReplay.SnapProtect (2033 ms)
2016-07-10T18:10:40.215 INFO:tasks.workunit.client.0.clara005.stdout:[ RUN      ] TestJournalReplay.SnapUnprotect
2016-07-10T18:10:42.243 INFO:tasks.workunit.client.0.clara005.stdout:[       OK ] TestJournalReplay.SnapUnprotect (2029 ms)
2016-07-10T18:10:42.244 INFO:tasks.workunit.client.0.clara005.stdout:[ RUN      ] TestJournalReplay.SnapRename
2016-07-10T18:10:43.360 INFO:tasks.workunit.client.0.clara005.stdout:[       OK ] TestJournalReplay.SnapRename (1117 ms)
2016-07-10T18:10:43.361 INFO:tasks.workunit.client.0.clara005.stdout:[ RUN      ] TestJournalReplay.SnapRollback
2016-07-10T18:10:44.366 INFO:tasks.workunit.client.0.clara005.stdout:[       OK ] TestJournalReplay.SnapRollback (1006 ms)
2016-07-10T18:10:44.367 INFO:tasks.workunit.client.0.clara005.stdout:[ RUN      ] TestJournalReplay.SnapRemove
2016-07-10T18:10:48.255 INFO:tasks.workunit.client.0.clara005.stdout:[       OK ] TestJournalReplay.SnapRemove (3889 ms)
2016-07-10T18:10:48.256 INFO:tasks.workunit.client.0.clara005.stdout:[ RUN      ] TestJournalReplay.Rename
2016-07-10T18:10:48.459 INFO:tasks.workunit.client.0.clara005.stdout:[       OK ] TestJournalReplay.Rename (205 ms)
2016-07-10T18:10:48.461 INFO:tasks.workunit.client.0.clara005.stdout:[ RUN      ] TestJournalReplay.Resize
2016-07-10T18:10:48.647 INFO:tasks.workunit.client.0.clara005.stdout:[       OK ] TestJournalReplay.Resize (187 ms)
2016-07-10T18:10:48.648 INFO:tasks.workunit.client.0.clara005.stdout:[ RUN      ] TestJournalReplay.Flatten
2016-07-10T18:10:49.556 INFO:tasks.workunit.client.0.clara005.stdout:[       OK ] TestJournalReplay.Flatten (909 ms)
2016-07-10T18:10:49.557 INFO:tasks.workunit.client.0.clara005.stdout:[ RUN      ] TestJournalReplay.ObjectPosition
2016-07-10T18:10:49.727 INFO:tasks.workunit.client.0.clara005.stdout:[       OK ] TestJournalReplay.ObjectPosition (171 ms)
2016-07-10T18:10:50.245 INFO:tasks.workunit.client.0.clara005.stdout:[----------] 13 tests from TestJournalReplay (17622 ms total)
2016-07-10T18:10:50.246 INFO:tasks.workunit.client.0.clara005.stdout:
2016-07-10T18:10:50.247 INFO:tasks.workunit.client.0.clara005.stdout:[----------] Global test environment tear-down
2016-07-10T18:10:50.248 INFO:tasks.workunit.client.0.clara005.stdout:[==========] 166 tests from 10 test cases ran. (326152 ms total)
2016-07-10T18:10:50.249 INFO:tasks.workunit.client.0.clara005.stdout:[  PASSED  ] 165 tests.
2016-07-10T18:10:50.250 INFO:tasks.workunit.client.0.clara005.stdout:[  FAILED  ] 1 test, listed below:
2016-07-10T18:10:50.251 INFO:tasks.workunit.client.0.clara005.stdout:[  FAILED  ] TestLibRBD.SnapCreateViaLockOwner


Version-Release number of selected component (if applicable):
10.2.2-15.el7cp (60cd52496ca02bdde9c2f4191e617f75166d87b6)

How reproducible:
1/1


Steps to Reproduce:
1. Run rbd/librbd/{cache/writeback.yaml cachepool/small.yaml clusters/{fixed-3.yaml openstack.yaml} copy-on-read/off.yaml fs/xfs.yaml msgr-failures/few.yaml workloads/c_api_tests_with_journaling.yaml


Additional info:

Full logs:
http://magna002.ceph.redhat.com/vasu-2016-07-09_20:45:55-rbd-jewel---basic-multi/244463/teuthology.log

Comment 2 Jason Dillaman 2016-07-11 17:56:29 UTC
@Vasu: please include the full output from the fail test.

Comment 3 Vasu Kulkarni 2016-07-11 18:01:40 UTC
@Jason,

Do you want to use this full logs from here instead, its too many lines?


Full logs:
http://magna002.ceph.redhat.com/vasu-2016-07-09_20:45:55-rbd-jewel---basic-multi/244463/teuthology.log

Comment 4 Jason Dillaman 2016-07-11 18:21:35 UTC
@Vasu: the test case will output what line had a failure -- you cut out that detail from the test's console output.

Comment 5 Vasu Kulkarni 2016-07-11 18:28:49 UTC
@Jason,

below is the line
2016-07-10T18:06:27.771 INFO:tasks.workunit.client.0.clara005.stdout:[ RUN      ] TestLibRBD.SnapCreateViaLockOwner
2016-07-10T18:06:27.772 INFO:tasks.workunit.client.0.clara005.stdout:using new format!
2016-07-10T18:06:28.250 INFO:tasks.workunit.client.0.clara005.stdout:test/librbd/test_librbd.cc:3094: Failure

Comment 11 Vasu Kulkarni 2016-07-19 19:07:28 UTC
Verified in 10.2.2-22

Comment 13 errata-xmlrpc 2016-08-23 19:44:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1755.html