1347174 – Object Map and Fast Diff Flag is getting invalid, while doing multiple times enable/disable of Journaling on an Image

Bug 1347174 - Object Map and Fast Diff Flag is getting invalid, while doing multiple times enable/disable of Journaling on an Image

Summary: Object Map and Fast Diff Flag is getting invalid, while doing multiple times ...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Ceph Storage
Classification:	Red Hat Storage
Component:	RBD
Sub Component:
Version:	2.0
Hardware:	x86_64
OS:	Linux
Priority:	medium
Severity:	high
Target Milestone:	rc
Target Release:	2.1
Assignee:	Jason Dillaman
QA Contact:	Rachana Patel
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2016-06-16 08:41 UTC by Tanay Ganguly
Modified:	2017-07-30 15:25 UTC (History)
CC List:	7 users (show)
Fixed In Version:	RHEL: ceph-10.2.3-2.el7cp Ubuntu: ceph_10.2.3-3redhat1xenial
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2016-11-22 19:26:33 UTC
Embargoed:
Dependent Products:
Flags:	hyelloji: needinfo+

Attachments	(Terms of Use)
Master Log (2.38 MB, text/plain) 2016-06-16 08:41 UTC, Tanay Ganguly	no flags	Details
Slave Log (13.58 KB, text/plain) 2016-06-16 08:42 UTC, Tanay Ganguly	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Ceph Project Bug Tracker	16350	0	None	None	None	2016-06-16 13:59:10 UTC
Red Hat Product Errata	RHSA-2016:2815	0	normal	SHIPPED_LIVE	Moderate: Red Hat Ceph Storage security, bug fix, and enhancement update	2017-03-22 02:06:33 UTC

Description Tanay Ganguly 2016-06-16 08:41:34 UTC

Created attachment 1168632 [details]
Master Log

Description of problem:
object map invalid, fast diff invalid

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Create an Image with Journaling enabled.
2. Start IO using bench-write.
3. Then execute:

for i in {1..10}; do rbd feature disable RBD/test1 journaling --cluster master; rbd bench-write RBD/test1 --io-size 10240 --cluster master; rbd feature enable RBD/test1 journaling --cluster master; done

Master:
rbd image 'test1':
        size 10240 MB in 2560 objects
        order 22 (4096 kB objects)
        block_name_prefix: rbd_data.11052ae8944a
        format: 2
        features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling
        flags: object map invalid, fast diff invalid
        journal: 11052ae8944a
        mirroring state: enabled
        mirroring global id: d415bdfd-6a78-4ae5-a651-33437011bbb0
        mirroring primary: true


Slave:
rbd image 'test1':
        size 10240 MB in 2560 objects
        order 22 (4096 kB objects)
        block_name_prefix: rbd_data.10783837f1df
        format: 2
        features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling
        flags: 
        journal: 10783837f1df
        mirroring state: enabled
        mirroring global id: d415bdfd-6a78-4ae5-a651-33437011bbb0
        mirroring primary: false


Actual results:
I am seeing Object Map and fast diff getting invalid on the Master Node.

Expected results:
Flag should not get invalid.

Additional info:
Master Log
There were no crash.

-------------------------------------------------------------------------------

Master:

systemctl status -l ceph-rbd-mirror@master
● ceph-rbd-mirror - Ceph rbd mirror daemon
   Loaded: loaded (/usr/lib/systemd/system/ceph-rbd-mirror@.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2016-06-16 11:15:02 IST; 2h 52min ago
 Main PID: 66591 (rbd-mirror)
   CGroup: /system.slice/system-ceph\x2drbd\x2dmirror.slice/ceph-rbd-mirror
           └─66591 /usr/bin/rbd-mirror -f --cluster master --id master --setuser ceph --setgroup ceph

Jun 16 13:56:24 x.x rbd-mirror[66591]: 2016-06-16 13:56:24.254626 7f702f7fe700 -1 JournalMetadata: operator(): failed to watch journal(2) No such file or directory
Jun 16 13:56:24 x.x rbd-mirror[66591]: 2016-06-16 13:56:24.254646 7f702f7fe700 -1 JournalMetadata: failed to initialize immutable metadata: (2) No such file or directory
Jun 16 13:56:54 x.x rbd-mirror[66591]: 2016-06-16 13:56:54.245727 7f702f7fe700 -1 JournalMetadata: operator(): failed to watch journal(2) No such file or directory
Jun 16 13:56:54 x.x rbd-mirror[66591]: 2016-06-16 13:56:54.245738 7f702f7fe700 -1 JournalMetadata: failed to initialize immutable metadata: (2) No such file or directory
Jun 16 13:58:27 x.x rbd-mirror[66591]: 2016-06-16 13:58:27.540204 7f701cff9700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x7f6fe8004b30 handle_get_remote_tag_class: failed to retrieve remote client: (2) No such file or directory
Jun 16 13:58:27 x.x rbd-mirror[66591]: 2016-06-16 13:58:27.540278 7f70508ee700 -1 rbd::mirror::ImageReplayer: 0x7f6fe8001ea0 [1/659a83ea-3ead-4cc1-8b5b-8c1ffb4f327b] operator(): start failed: (2) No such file or directory
Jun 16 13:58:27 x.x rbd-mirror[66591]: 2016-06-16 13:58:27.552084 7f702f7fe700 -1 JournalMetadata: operator(): failed to watch journal(2) No such file or directory
Jun 16 13:58:27 x.x rbd-mirror[66591]: 2016-06-16 13:58:27.552103 7f702f7fe700 -1 JournalMetadata: failed to initialize immutable metadata: (2) No such file or directory
Jun 16 13:58:57 x.x rbd-mirror[66591]: 2016-06-16 13:58:57.529338 7f702f7fe700 -1 JournalMetadata: operator(): failed to watch journal(2) No such file or directory
Jun 16 13:58:57 x.x rbd-mirror[66591]: 2016-06-16 13:58:57.529359 7f702f7fe700 -1 JournalMetadata: failed to initialize immutable metadata: (2) No such file or directory




Slave:

systemctl status -l ceph-rbd-mirror@slave
● ceph-rbd-mirror - Ceph rbd mirror daemon
   Loaded: loaded (/usr/lib/systemd/system/ceph-rbd-mirror@.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2016-06-15 16:53:29 UTC; 15h ago
 Main PID: 30993 (rbd-mirror)
   CGroup: /system.slice/system-ceph\x2drbd\x2dmirror.slice/ceph-rbd-mirror
           └─30993 /usr/bin/rbd-mirror -f --cluster slave --id slave --setuser ceph --setgroup ceph

Jun 16 08:28:12 magna003 rbd-mirror[30993]: 2016-06-16 08:28:12.139102 7f8d41ffb700 -1 rbd::mirror::image_sync::ImageCopyRequest: 0x7f8cf000bab0 handle_object_copy: object copy failed: (2) No such file or directory
Jun 16 08:28:22 magna003 rbd-mirror[30993]: 2016-06-16 08:28:22.573271 7f8d41ffb700 -1 rbd::mirror::image_sync::ObjectCopyRequest: 0x7f8d04104100 handle_read_object: failed to read from remote object: (2) No such file or directory
Jun 16 08:28:22 magna003 rbd-mirror[30993]: 2016-06-16 08:28:22.573285 7f8d41ffb700 -1 rbd::mirror::image_sync::ImageCopyRequest: 0x7f8cf000bab0 handle_object_copy: object copy failed: (2) No such file or directory
Jun 16 08:28:22 magna003 rbd-mirror[30993]: 2016-06-16 08:28:22.573293 7f8d41ffb700 -1 rbd::mirror::ImageSync: 0x7f8cf0056210 handle_copy_image: failed to copy image: (2) No such file or directory
Jun 16 08:28:22 magna003 rbd-mirror[30993]: 2016-06-16 08:28:22.573297 7f8d41ffb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x7f8d0002a330 handle_image_sync: failed to sync remote image: (2) No such file or directory
Jun 16 08:28:22 magna003 rbd-mirror[30993]: 2016-06-16 08:28:22.883709 7f8d6981b700 -1 rbd::mirror::ImageReplayer: 0x7f8d000065c0 [1/659a83ea-3ead-4cc1-8b5b-8c1ffb4f327b] operator(): start failed: (2) No such file or directory
Jun 16 08:28:34 magna003 rbd-mirror[30993]: 2016-06-16 08:28:34.176266 7f8d41ffb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x7f8d00010e50 handle_get_remote_tag_class: failed to retrieve remote client: (2) No such file or directory
Jun 16 08:28:34 magna003 rbd-mirror[30993]: 2016-06-16 08:28:34.176330 7f8d6981b700 -1 rbd::mirror::ImageReplayer: 0x7f8d000065c0 [1/659a83ea-3ead-4cc1-8b5b-8c1ffb4f327b] operator(): start failed: (2) No such file or directory
Jun 16 08:29:04 magna003 rbd-mirror[30993]: 2016-06-16 08:29:04.175564 7f8d58ef8700 -1 JournalMetadata: operator(): failed to watch journal(2) No such file or directory
Jun 16 08:29:04 magna003 rbd-mirror[30993]: 2016-06-16 08:29:04.175584 7f8d58ef8700 -1 JournalMetadata: failed to initialize immutable metadata: (2) No such file or directory

Comment 2 Tanay Ganguly 2016-06-16 08:42:29 UTC

Created attachment 1168635 [details]
Slave Log

Comment 3 Tanay Ganguly 2016-06-16 09:06:51 UTC

Version:
ceph version 10.2.2-1.el7cp
rbd-mirror-10.2.2-1.el7cp.x86_64

Reproduced it Once.

Comment 10 Jason Dillaman 2016-08-10 19:16:52 UTC

Upstream pull request: https://github.com/ceph/ceph/pull/9899

Comment 11 kiran raje urs J 2016-09-14 09:39:17 UTC

changing the need info from Tanay to Hemanth.

Comment 16 Rachana Patel 2016-10-21 19:34:50 UTC

verified with 10.2.3-8.el7cp.x86_64.

Followed procedure mentioned in bug description. no error messages hence moving to verified

Comment 18 errata-xmlrpc 2016-11-22 19:26:33 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2815.html

Note You need to log in before you can comment on or make changes to this bug.