1326882 – [RBD] rbd cores when journaling is disabled during bench-write

Bug 1326882 - [RBD] rbd cores when journaling is disabled during bench-write

Summary: [RBD] rbd cores when journaling is disabled during bench-write

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Ceph Storage
Classification:	Red Hat Storage
Component:	RBD
Sub Component:
Version:	2.0
Hardware:	Unspecified
OS:	Linux
Priority:	unspecified
Severity:	urgent
Target Milestone:	rc
Target Release:	2.0
Assignee:	Jason Dillaman
QA Contact:	ceph-qe-bugs
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2016-04-13 16:01 UTC by Tejas
Modified:	2017-07-30 15:35 UTC (History)
CC List:	6 users (show)
Fixed In Version:	ceph-10.2.0-1.el7cp
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2016-08-23 19:36:04 UTC
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Ceph Project Bug Tracker	15487	0	None	None	None	2016-04-13 16:08:51 UTC
Red Hat Product Errata	RHBA-2016:1755	0	normal	SHIPPED_LIVE	Red Hat Ceph Storage 2.0 bug fix and enhancement update	2016-08-23 23:23:52 UTC

Description Tejas 2016-04-13 16:01:57 UTC

Description of problem:
if journaling feature is disabled when writes are happening on an image, bench-write cores.

Version-Release number of selected component (if applicable):
ceph 10.1.1

How reproducible:
Always

Steps to Reproduce:
1. create an rbd image with layering, exclusive-lock,fast-diff, object-map, jounaling
2. start bench-write on the image.
3. when writes are happening disable the journaling feature.

Actual results:
rbd cores

Expected results:
Bench-write should handle the writes, or if the scenario is not valid, it should exit gracefully

I am not able to attach the core dump since its  650 M. Let me know if I can place it somewhere.

Additional info:


[root@magna009 ~]# rbd feature enable Tejas/img1 object-map,fast-diff,exclusive-lock,journaling
[root@magna009 ~]# 
[root@magna009 ~]# 
[root@magna009 ~]# rbd info Tejas/img1
rbd image 'img1':
	size 10240 MB in 2560 objects
	order 22 (4096 kB objects)
	block_name_prefix: rbd_data.1060238e1f29
	format: 2
	features: layering, exclusive-lock, object-map, fast-diff, journaling
	flags: object map invalid, fast diff invalid
	journal: 1060238e1f29
	mirroring state: disabled
[root@magna009 ~]# 
[root@magna009 ~]# 
[root@magna009 ~]# rbd feature disable Tejas/img1 fast-diff
[root@magna009 ~]# 
[root@magna009 ~]# 
[root@magna009 ~]# rbd du Tejas/img1
warning: fast-diff map is not enabled for img1. operation may be slow.
NAME PROVISIONED USED 
img1      10240M    0 
[root@magna009 ~]# 
[root@magna009 ~]# rbd bench-write Tejas/img1
bench-write  io_size 4096 io_threads 16 bytes 1073741824 pattern sequential
  SEC       OPS   OPS/SEC   BYTES/SEC
    1      1302   1312.95  5377831.11
    2      1571    792.73  3247012.59
    3      1807    607.12  2486762.87
    4      2101    528.48  2164651.42
    5      2347    472.16  1933961.84
    6      2535    246.77  1010757.22
    7      2630    211.50  866302.34
    8      2726    183.81  752870.01
    9      2820    143.91  589467.27
   10      2922    114.90  470631.65
   11      3077    108.37  443889.43
   12      3263    126.61  518581.57
   13      3479    150.55  616656.14
2016-04-13 14:13:54.758371 7fd6bd5df700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
   14      3653    166.62  682461.72
2016-04-13 14:13:54.890939 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-04-13 14:13:54.994015 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-04-13 14:13:55.103043 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-04-13 14:13:55.204755 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-04-13 14:13:55.322081 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-04-13 14:13:55.432231 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-04-13 14:13:55.554111 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-04-13 14:13:55.656456 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-04-13 14:13:55.765681 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-04-13 14:13:55.889791 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-04-13 14:13:55.991108 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-04-13 14:13:56.109197 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-04-13 14:13:56.211776 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-04-13 14:13:56.316547 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-04-13 14:13:56.423027 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-04-13 14:13:56.524714 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-04-13 14:13:56.626742 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-04-13 14:13:56.735933 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-04-13 14:13:56.866603 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-04-13 14:13:56.980940 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-04-13 14:13:57.082541 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
librbd/LibrbdWriteback.cc: In function 'void librbd::C_WriteJournalCommit::commit_io_event_extent(int)' thread 7fd6bd5df700 time 2016-04-13 14:13:57.116138
librbd/LibrbdWriteback.cc: 143: FAILED assert(image_ctx->journal != __null)
 ceph version 10.1.1-1.el7cp (61adb020219fbad4508050b5f0a792246ba74dae)
 1: (()+0x273e75) [0x7fd6d3271e75]
 2: (()+0xec5e4) [0x7fd6d30ea5e4]
 3: (()+0xeca32) [0x7fd6d30eaa32]
 4: (()+0x815d7) [0x7fd6d307f5d7]
 5: (()+0x9d00d) [0x7fd6c992200d]
 6: (()+0x85529) [0x7fd6c990a529]
 7: (()+0x16eb46) [0x7fd6c99f3b46]
 8: (()+0x7dc5) [0x7fd6c910cdc5]
 9: (clone()+0x6d) [0x7fd6c722228d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
*** Caught signal (Aborted) **
 in thread 7fd6bd5df700 thread_name:fn_anonymous
 ceph version 10.1.1-1.el7cp (61adb020219fbad4508050b5f0a792246ba74dae)
 1: (()+0x1d8dea) [0x7fd6dcd17dea]
 2: (()+0xf100) [0x7fd6c9114100]
 3: (gsignal()+0x37) [0x7fd6c71615f7]
 4: (abort()+0x148) [0x7fd6c7162ce8]
 5: (()+0x274057) [0x7fd6d3272057]
 6: (()+0xec5e4) [0x7fd6d30ea5e4]
 7: (()+0xeca32) [0x7fd6d30eaa32]
 8: (()+0x815d7) [0x7fd6d307f5d7]
 9: (()+0x9d00d) [0x7fd6c992200d]
 10: (()+0x85529) [0x7fd6c990a529]
 11: (()+0x16eb46) [0x7fd6c99f3b46]
 12: (()+0x7dc5) [0x7fd6c910cdc5]
 13: (clone()+0x6d) [0x7fd6c722228d]
2016-04-13 14:13:57.116631 7fd6bd5df700 -1 *** Caught signal (Aborted) **
 in thread 7fd6bd5df700 thread_name:fn_anonymous

 ceph version 10.1.1-1.el7cp (61adb020219fbad4508050b5f0a792246ba74dae)
 1: (()+0x1d8dea) [0x7fd6dcd17dea]
 2: (()+0xf100) [0x7fd6c9114100]
 3: (gsignal()+0x37) [0x7fd6c71615f7]
 4: (abort()+0x148) [0x7fd6c7162ce8]
 5: (()+0x274057) [0x7fd6d3272057]
 6: (()+0xec5e4) [0x7fd6d30ea5e4]
 7: (()+0xeca32) [0x7fd6d30eaa32]
 8: (()+0x815d7) [0x7fd6d307f5d7]
 9: (()+0x9d00d) [0x7fd6c992200d]
 10: (()+0x85529) [0x7fd6c990a529]
 11: (()+0x16eb46) [0x7fd6c99f3b46]
 12: (()+0x7dc5) [0x7fd6c910cdc5]
 13: (clone()+0x6d) [0x7fd6c722228d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
   -36> 2016-04-13 14:13:40.739184 7fd6dcb05d80  5 asok(0x7fd6e69f8e00) register_command perfcounters_dump hook 0x7fd6e69fdb90
   -35> 2016-04-13 14:13:40.739195 7fd6dcb05d80  5 asok(0x7fd6e69f8e00) register_command 1 hook 0x7fd6e69fdb90
   -34> 2016-04-13 14:13:40.739201 7fd6dcb05d80  5 asok(0x7fd6e69f8e00) register_command perf dump hook 0x7fd6e69fdb90
   -33> 2016-04-13 14:13:40.739206 7fd6dcb05d80  5 asok(0x7fd6e69f8e00) register_command perfcounters_schema hook 0x7fd6e69fdb90
   -32> 2016-04-13 14:13:40.739212 7fd6dcb05d80  5 asok(0x7fd6e69f8e00) register_command 2 hook 0x7fd6e69fdb90
   -31> 2016-04-13 14:13:40.739217 7fd6dcb05d80  5 asok(0x7fd6e69f8e00) register_command perf schema hook 0x7fd6e69fdb90
   -30> 2016-04-13 14:13:40.739223 7fd6dcb05d80  5 asok(0x7fd6e69f8e00) register_command perf reset hook 0x7fd6e69fdb90
   -29> 2016-04-13 14:13:40.739227 7fd6dcb05d80  5 asok(0x7fd6e69f8e00) register_command config show hook 0x7fd6e69fdb90
   -28> 2016-04-13 14:13:40.739233 7fd6dcb05d80  5 asok(0x7fd6e69f8e00) register_command config set hook 0x7fd6e69fdb90
   -27> 2016-04-13 14:13:40.739240 7fd6dcb05d80  5 asok(0x7fd6e69f8e00) register_command config get hook 0x7fd6e69fdb90
   -26> 2016-04-13 14:13:40.739245 7fd6dcb05d80  5 asok(0x7fd6e69f8e00) register_command config diff hook 0x7fd6e69fdb90
   -25> 2016-04-13 14:13:40.739250 7fd6dcb05d80  5 asok(0x7fd6e69f8e00) register_command log flush hook 0x7fd6e69fdb90
   -24> 2016-04-13 14:13:40.739254 7fd6dcb05d80  5 asok(0x7fd6e69f8e00) register_command log dump hook 0x7fd6e69fdb90
   -23> 2016-04-13 14:13:40.739260 7fd6dcb05d80  5 asok(0x7fd6e69f8e00) register_command log reopen hook 0x7fd6e69fdb90
   -22> 2016-04-13 14:13:54.758371 7fd6bd5df700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
   -21> 2016-04-13 14:13:54.890939 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
   -20> 2016-04-13 14:13:54.994015 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
   -19> 2016-04-13 14:13:55.103043 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
   -18> 2016-04-13 14:13:55.204755 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
   -17> 2016-04-13 14:13:55.322081 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
   -16> 2016-04-13 14:13:55.432231 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
   -15> 2016-04-13 14:13:55.554111 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
   -14> 2016-04-13 14:13:55.656456 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
   -13> 2016-04-13 14:13:55.765681 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
   -12> 2016-04-13 14:13:55.889791 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
   -11> 2016-04-13 14:13:55.991108 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
   -10> 2016-04-13 14:13:56.109197 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
    -9> 2016-04-13 14:13:56.211776 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
    -8> 2016-04-13 14:13:56.316547 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
    -7> 2016-04-13 14:13:56.423027 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
    -6> 2016-04-13 14:13:56.524714 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
    -5> 2016-04-13 14:13:56.626742 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
    -4> 2016-04-13 14:13:56.735933 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
    -3> 2016-04-13 14:13:56.866603 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
    -2> 2016-04-13 14:13:56.980940 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
    -1> 2016-04-13 14:13:57.082541 7fd6956f9700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
     0> 2016-04-13 14:13:57.116631 7fd6bd5df700 -1 *** Caught signal (Aborted) **
 in thread 7fd6bd5df700 thread_name:fn_anonymous

 ceph version 10.1.1-1.el7cp (61adb020219fbad4508050b5f0a792246ba74dae)
 1: (()+0x1d8dea) [0x7fd6dcd17dea]
 2: (()+0xf100) [0x7fd6c9114100]
 3: (gsignal()+0x37) [0x7fd6c71615f7]
 4: (abort()+0x148) [0x7fd6c7162ce8]
 5: (()+0x274057) [0x7fd6d3272057]
 6: (()+0xec5e4) [0x7fd6d30ea5e4]
 7: (()+0xeca32) [0x7fd6d30eaa32]
 8: (()+0x815d7) [0x7fd6d307f5d7]
 9: (()+0x9d00d) [0x7fd6c992200d]
 10: (()+0x85529) [0x7fd6c990a529]
 11: (()+0x16eb46) [0x7fd6c99f3b46]
 12: (()+0x7dc5) [0x7fd6c910cdc5]
 13: (clone()+0x6d) [0x7fd6c722228d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 0 lockdep
   0/ 0 context
   0/ 0 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 0 buffer
   0/ 0 timer
   0/ 0 filer
   0/ 1 striper
   0/ 0 objecter
   0/ 0 rados
   0/ 0 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 0 journaler
   0/ 5 objectcacher
   0/ 0 client
   0/ 0 osd
   0/ 0 optracker
   0/ 0 objclass
   0/ 0 filestore
   0/ 0 journal
   0/ 0 ms
   0/ 0 mon
   0/ 0 monc
   0/ 0 paxos
   0/ 0 tp
   0/ 0 auth
   1/ 5 crypto
   0/ 0 finisher
   0/ 0 heartbeatmap
   0/ 0 perfcounter
   0/ 0 rgw
   1/10 civetweb
   1/ 5 javaclient
   0/ 0 asok
   0/ 0 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 newstore
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   1/ 5 kinetic
   1/ 5 fuse
  -2/-2 (syslog threshold)
  99/99 (stderr threshold)
  max_recent       500
  max_new         1000
  log_file /var/log/rbd-clients//qemu-guest-9968.log
--- end dump of recent events ---
Aborted (core dumped)
[root@magna009 ~]# 
[root@magna009 ~]# 
[root@magna009 ~]# 
[root@magna009 ~]# ls /var/crash/
[root@magna009 ~]# ls
anaconda-ks.cfg  cobbler.ks  core.9968  ks-post.log  ks-post-nochroot.log  ks-pre.log

Comment 2 Jason Dillaman 2016-04-13 16:08:52 UTC

Upstream PR: https://github.com/ceph/ceph/pull/8511

Comment 3 Ken Dreyer (Red Hat) 2016-04-26 20:53:20 UTC

The above PR is present in v10.2.0.

Comment 5 Tanay Ganguly 2016-05-31 07:15:01 UTC

Re-opening this Ticket.
ceph version 10.2.1-6.el7cp 

As i am still seeing lots of error message, when bench-write is running in parallel with enable/disable of Journaling


-----------------------------------------------------------------------------

# rbd bench-write -p cephfs_data --image img1 --io-size 10240 --io-pattern rand
bench-write  io_size 10240 io_threads 16 bytes 1073741824 pattern random
  SEC       OPS   OPS/SEC   BYTES/SEC



2016-05-31 07:03:24.530310 7f0589efa700 -1 JournalMetadata: init: failed to watch journal(2) No such file or directory
2016-05-31 07:03:24.530368 7f05896f9700 -1 librbd::Journal: 0x7f057000a640 handle_initialized: failed to initialize journal: (2) No such file or directory
2016-05-31 07:03:24.530421 7f05896f9700 -1 librbd::exclusive_lock::AcquireRequest: failed to open journal: (2) No such file or directory
2016-05-31 07:03:24.530431 7f05896f9700 -1 librbd::exclusive_lock::AcquireRequest: failed to close journal: (2) No such file or directory
2016-05-31 07:03:24.542757 7f05896f9700 -1 librbd::ExclusiveLock: failed to acquire exclusive lock:(2) No such file or directory
2016-05-31 07:03:24.554547 7f0589efa700 -1 JournalMetadata: init: failed to watch journal(2) No such file or directory
2016-05-31 07:03:24.554593 7f05896f9700 -1 librbd::Journal: 0x7f057000a640 handle_initialized: failed to initialize journal: (2) No such file or directory
2016-05-31 07:03:24.554642 7f05896f9700 -1 librbd::exclusive_lock::AcquireRequest: failed to open journal: (2) No such file or directory
2016-05-31 07:03:24.554650 7f05896f9700 -1 librbd::exclusive_lock::AcquireRequest: failed to close journal: (2) No such file or directory
2016-05-31 07:03:24.560040 7f05896f9700 -1 librbd::ExclusiveLock: failed to acquire exclusive lock:(2) No such file or directory
2016-05-31 07:03:24.571100 7f0589efa700 -1 JournalMetadata: init: failed to watch journal(2) No such file or directory
2016-05-31 07:03:24.571151 7f05896f9700 -1 librbd::Journal: 0x7f057000a640 handle_initialized: failed to initialize journal: (2) No such file or directory
2016-05-31 07:03:24.571200 7f05896f9700 -1 librbd::exclusive_lock::AcquireRequest: failed to open journal: (2) No such file or directory
2016-05-31 07:03:24.571208 7f05896f9700 -1 librbd::exclusive_lock::AcquireRequest: failed to close journal: (2) No such file or directory
2016-05-31 07:03:24.576809 7f05896f9700 -1 librbd::ExclusiveLock: failed to acquire exclusive lock:(2) No such file or directory
2016-05-31 07:03:24.589047 7f0589efa700 -1 JournalMetadata: init: failed to watch journal(2) No such file or directory
2016-05-31 07:03:24.589097 7f05896f9700 -1 librbd::Journal: 0x7f057000a640 handle_initialized: failed to initialize journal: (2) No such file or directory
2016-05-31 07:03:24.589143 7f05896f9700 -1 librbd::exclusive_lock::AcquireRequest: failed to open journal: (2) No such file or directory
2016-05-31 07:03:24.589151 7f05896f9700 -1 librbd::exclusive_lock::AcquireRequest: failed to close journal: (2) No such file or directory
2016-05-31 07:03:24.602635 7f05896f9700 -1 librbd::ExclusiveLock: failed to acquire exclusive lock:(2) No such file or directory
2016-05-31 07:03:24.636691 7f0589efa700 -1 JournalMetadata: failed to locate client:
2016-05-31 07:03:24.636710 7f05896f9700 -1 librbd::Journal: 0x7f057000a640 handle_initialized: failed to initialize journal: (2) No such file or directory
2016-05-31 07:03:24.652423 7f05896f9700 -1 librbd::exclusive_lock::AcquireRequest: failed to open journal: (2) No such file or directory
2016-05-31 07:03:24.652434 7f05896f9700 -1 librbd::exclusive_lock::AcquireRequest: failed to close journal: (2) No such file or directory
2016-05-31 07:03:24.669508 7f05896f9700 -1 librbd::ExclusiveLock: failed to acquire exclusive lock:(2) No such file or directory
   37         1      0.45   4595.82
   38      1047     27.97  286434.85
   39      1212     31.48  322309.09
   40      1357     34.32  351454.88
2016-05-31 07:03:37.119280 7f0589efa700 -1 JournalMetadata: init: failed to watch journal(2) No such file or directory
2016-05-31 07:03:37.119302 7f05896f9700 -1 librbd::Journal: 0x7f0570008b20 handle_initialized: failed to initialize journal: (2) No such file or directory
2016-05-31 07:03:37.119337 7f05896f9700 -1 librbd::exclusive_lock::AcquireRequest: failed to open journal: (2) No such file or directory
2016-05-31 07:03:37.119343 7f05896f9700 -1 librbd::exclusive_lock::AcquireRequest: failed to close journal: (2) No such file or directory
2016-05-31 07:03:37.133198 7f05896f9700 -1 librbd::ExclusiveLock: failed to acquire exclusive lock:(2) No such file or directory


-----------------------------------------------------------------------------


for i in {1..200}; do rbd feature enable cephfs_data/img1 journaling; sleep 2; rbd feature disable cephfs_data/img1 journaling; done


2016-05-31 07:03:24.417365 7f8574ff9700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:03:37.007677 7fa6677fe700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:03:49.975260 7fc307fff700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:04:05.286831 7f8a6bfff700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:04:05.388188 7f8a37fff700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-05-31 07:04:05.489165 7f8a37fff700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-05-31 07:04:05.590001 7f8a37fff700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-05-31 07:04:21.811341 7f0b7e7fc700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:04:21.912928 7f0b567fc700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-05-31 07:04:34.691536 7eff4ffff700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:04:50.387203 7fbd117fa700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:05:06.553524 7fbe8affd700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:05:21.178626 7fd4eeffd700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:05:34.122701 7effd77fe700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:05:48.703560 7fc04e7fc700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:06:03.779298 7fe744ff9700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:06:18.031368 7fa9a3fff700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:06:32.759689 7fe71cff9700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:06:45.695572 7f30e0ff9700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:06:58.470836 7f418a7fc700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:07:10.868918 7f74d37fe700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:07:25.661293 7f24776fd700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:07:39.784748 7fae3d7fa700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:07:54.513960 7f4658ff9700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:08:08.900208 7f7e1d7fa700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:08:20.790760 7f40d3fff700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:08:33.231866 7f46cdffb700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:08:48.871778 7fc6c4ef8700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:09:02.943667 7f95ea7fc700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected
2016-05-31 07:09:03.044967 7f95c27fc700 -1 JournalMetadata: handle_watch_reset: failed to watch journal(2) No such file or directory
2016-05-31 07:09:15.367820 7fbd526fb700 -1 JournalMetadata: journal watch error: (107) Transport endpoint is not connected

Comment 8 Tanay Ganguly 2016-06-01 10:46:04 UTC

Marking this BUG as Verified as i am not seeing the Crash.
ceph version 10.2.1-6

I have filed a new ticket for the issue mention in Comment5 
https://bugzilla.redhat.com/show_bug.cgi?id=1341604

Comment 10 errata-xmlrpc 2016-08-23 19:36:04 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1755.html

Note You need to log in before you can comment on or make changes to this bug.