Bug 1341604

Summary: Seeing lots of error message, when bench-write is running in parallel with enable/disable of Journaling
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Tanay Ganguly <tganguly>
Component: RBDAssignee: Jason Dillaman <jdillama>
Status: CLOSED ERRATA QA Contact: Tanay Ganguly <tganguly>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 2.0CC: ceph-eng-bugs, ceph-qe-bugs, gmeno, hnallurv, kdreyer, kurs, tchandra
Target Milestone: rcKeywords: Reopened
Target Release: 2.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: ceph-10.2.2-1.el7cp Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-08-23 19:40:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1343229    

Description Tanay Ganguly 2016-06-01 10:45:33 UTC
Description of problem:
Seeing lots of error message, when bench-write is running in parallel with enable/disable of Journaling

Version-Release number of selected component (if applicable):
ceph version 10.2.1-6.el7cp 

How reproducible:
2 Times

Steps to Reproduce:
1. create an rbd image with layering, exclusive-lock,fast-diff, object-map, jounaling
2. start bench-write on the image.
3. when writes are happening disable the journaling feature.

Actual results:
Seeing lots of Error Message

Expected results:
Error message should not be shown.

-----------------------------------------------------------------------------

# rbd bench-write -p cephfs_data --image img1 --io-size 10240 --io-pattern rand
bench-write  io_size 10240 io_threads 16 bytes 1073741824 pattern random
  SEC       OPS   OPS/SEC   BYTES/SEC



2016-05-31 07:03:24.530310 7f0589efa700 -1 JournalMetadata: init: failed to watch journal(2) No such file or directory
2016-05-31 07:03:24.530368 7f05896f9700 -1 librbd::Journal: 0x7f057000a640 handle_initialized: failed to initialize journal: (2) No such file or directory
2016-05-31 07:03:24.530421 7f05896f9700 -1 librbd::exclusive_lock::AcquireRequest: failed to open journal: (2) No such file or directory
2016-05-31 07:03:24.530431 7f05896f9700 -1 librbd::exclusive_lock::AcquireRequest: failed to close journal: (2) No such file or directory
2016-05-31 07:03:24.542757 7f05896f9700 -1 librbd::ExclusiveLock: failed to acquire exclusive lock:(2) No such file or directory
2016-05-31 07:03:24.554547 7f0589efa700 -1 JournalMetadata: init: failed to watch journal(2) No such file or directory
2016-05-31 07:03:24.554593 7f05896f9700 -1 librbd::Journal: 0x7f057000a640 handle_initialized: failed to initialize journal: (2) No such file or directory
2016-05-31 07:03:24.554642 7f05896f9700 -1 librbd::exclusive_lock::AcquireRequest: failed to open journal: (2) No such file or directory
2016-05-31 07:03:24.554650 7f05896f9700 -1 librbd::exclusive_lock::AcquireRequest: failed to close journal: (2) No such file or directory
2016-05-31 07:03:24.560040 7f05896f9700 -1 librbd::ExclusiveLock: failed to acquire exclusive lock:(2) No such file or directory
2016-05-31 07:03:24.571100 7f0589efa700 -1 JournalMetadata: init: failed to watch journal(2) No such file or directory
2016-05-31 07:03:24.571151 7f05896f9700 -1 librbd::Journal: 0x7f057000a640 handle_initialized: failed to initialize journal: (2) No such file or directory
2016-05-31 07:03:24.571200 7f05896f9700 -1 librbd::exclusive_lock::AcquireRequest: failed to open journal: (2) No such file or directory
2016-05-31 07:03:24.571208 7f05896f9700 -1 librbd::exclusive_lock::AcquireRequest: failed to close journal: (2) No such file or directory
2016-05-31 07:03:24.576809 7f05896f9700 -1 librbd::ExclusiveLock: failed to acquire exclusive lock:(2) No such file or directory
2016-05-31 07:03:24.589047 7f0589efa700 -1 JournalMetadata: init: failed to watch journal(2) No such file or directory
2016-05-31 07:03:24.589097 7f05896f9700 -1 librbd::Journal: 0x7f057000a640 handle_initialized: failed to initialize journal: (2) No such file or directory
2016-05-31 07:03:24.589143 7f05896f9700 -1 librbd::exclusive_lock::AcquireRequest: failed to open journal: (2) No such file or directory
2016-05-31 07:03:24.589151 7f05896f9700 -1 librbd::exclusive_lock::AcquireRequest: failed to close journal: (2) No such file or directory
2016-05-31 07:03:24.602635 7f05896f9700 -1 librbd::ExclusiveLock: failed to acquire exclusive lock:(2) No such file or directory
2016-05-31 07:03:24.636691 7f0589efa700 -1 JournalMetadata: failed to locate client:
2016-05-31 07:03:24.636710 7f05896f9700 -1 librbd::Journal: 0x7f057000a640 handle_initialized: failed to initialize journal: (2) No such file or directory
2016-05-31 07:03:24.652423 7f05896f9700 -1 librbd::exclusive_lock::AcquireRequest: failed to open journal: (2) No such file or directory
2016-05-31 07:03:24.652434 7f05896f9700 -1 librbd::exclusive_lock::AcquireRequest: failed to close journal: (2) No such file or directory
2016-05-31 07:03:24.669508 7f05896f9700 -1 librbd::ExclusiveLock: failed to acquire exclusive lock:(2) No such file or directory
   37         1      0.45   4595.82
   38      1047     27.97  286434.85
   39      1212     31.48  322309.09
   40      1357     34.32  351454.88
2016-05-31 07:03:37.119280 7f0589efa700 -1 JournalMetadata: init: failed to watch journal(2) No such file or directory
2016-05-31 07:03:37.119302 7f05896f9700 -1 librbd::Journal: 0x7f0570008b20 handle_initialized: failed to initialize journal: (2) No such file or directory
2016-05-31 07:03:37.119337 7f05896f9700 -1 librbd::exclusive_lock::AcquireRequest: failed to open journal: (2) No such file or directory
2016-05-31 07:03:37.119343 7f05896f9700 -1 librbd::exclusive_lock::AcquireRequest: failed to close journal: (2) No such file or directory
2016-05-31 07:03:37.133198 7f05896f9700 -1 librbd::ExclusiveLock: failed to acquire exclusive lock:(2) No such file or directory


-----------------------------------------------------------------------------
For the below log messages there is a BUG 1335032
https://bugzilla.redhat.com/show_bug.cgi?id=1335032



for i in {1..200}; do rbd feature enable cephfs_data/img1 journaling; sleep 2; rbd feature disable cephfs_data/img1 journaling; done


2016-05-31 07:03:24.417365 7f8574ff9700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:03:37.007677 7fa6677fe700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:03:49.975260 7fc307fff700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:04:05.286831 7f8a6bfff700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:04:05.388188 7f8a37fff700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-05-31 07:04:05.489165 7f8a37fff700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-05-31 07:04:05.590001 7f8a37fff700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-05-31 07:04:21.811341 7f0b7e7fc700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:04:21.912928 7f0b567fc700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-05-31 07:04:34.691536 7eff4ffff700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:04:50.387203 7fbd117fa700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:05:06.553524 7fbe8affd700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:05:21.178626 7fd4eeffd700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:05:34.122701 7effd77fe700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:05:48.703560 7fc04e7fc700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:06:03.779298 7fe744ff9700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:06:18.031368 7fa9a3fff700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:06:32.759689 7fe71cff9700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:06:45.695572 7f30e0ff9700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:06:58.470836 7f418a7fc700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:07:10.868918 7f74d37fe700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:07:25.661293 7f24776fd700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:07:39.784748 7fae3d7fa700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:07:54.513960 7f4658ff9700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:08:08.900208 7f7e1d7fa700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:08:20.790760 7f40d3fff700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:08:33.231866 7f46cdffb700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:08:48.871778 7fc6c4ef8700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:09:02.943667 7f95ea7fc700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:09:03.044967 7f95c27fc700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-05-31 07:09:15.367820 7fbd526fb700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected

Comment 2 Jason Dillaman 2016-06-01 22:30:25 UTC

*** This bug has been marked as a duplicate of bug 1335032 ***

Comment 5 Jason Dillaman 2016-06-02 04:49:09 UTC
Upstream PR: https://github.com/ceph/ceph/pull/9440

Comment 6 Jason Dillaman 2016-06-10 17:35:11 UTC
Merged Jewel PR: https://github.com/ceph/ceph/pull/9610

Comment 7 Ken Dreyer (Red Hat) 2016-06-14 16:22:40 UTC
We will take this change in as part of the rebase to ceph 10.2.2.

Comment 11 errata-xmlrpc 2016-08-23 19:40:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1755.html