Bug 1347174 - Object Map and Fast Diff Flag is getting invalid, while doing multiple times enable/disable of Journaling on an Image
Summary: Object Map and Fast Diff Flag is getting invalid, while doing multiple times ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RBD
Version: 2.0
Hardware: x86_64
OS: Linux
medium
high
Target Milestone: rc
: 2.1
Assignee: Jason Dillaman
QA Contact: Rachana Patel
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-06-16 08:41 UTC by Tanay Ganguly
Modified: 2017-07-30 15:25 UTC (History)
7 users (show)

Fixed In Version: RHEL: ceph-10.2.3-2.el7cp Ubuntu: ceph_10.2.3-3redhat1xenial
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-11-22 19:26:33 UTC
Target Upstream Version:
hyelloji: needinfo+


Attachments (Terms of Use)
Master Log (2.38 MB, text/plain)
2016-06-16 08:41 UTC, Tanay Ganguly
no flags Details
Slave Log (13.58 KB, text/plain)
2016-06-16 08:42 UTC, Tanay Ganguly
no flags Details


Links
System ID Priority Status Summary Last Updated
Ceph Project Bug Tracker 16350 None None None 2016-06-16 13:59:10 UTC
Red Hat Product Errata RHSA-2016:2815 normal SHIPPED_LIVE Moderate: Red Hat Ceph Storage security, bug fix, and enhancement update 2017-03-22 02:06:33 UTC

Description Tanay Ganguly 2016-06-16 08:41:34 UTC
Created attachment 1168632 [details]
Master Log

Description of problem:
object map invalid, fast diff invalid

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Create an Image with Journaling enabled.
2. Start IO using bench-write.
3. Then execute:

for i in {1..10}; do rbd feature disable RBD/test1 journaling --cluster master; rbd bench-write RBD/test1 --io-size 10240 --cluster master; rbd feature enable RBD/test1 journaling --cluster master; done

Master:
rbd image 'test1':
        size 10240 MB in 2560 objects
        order 22 (4096 kB objects)
        block_name_prefix: rbd_data.11052ae8944a
        format: 2
        features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling
        flags: object map invalid, fast diff invalid
        journal: 11052ae8944a
        mirroring state: enabled
        mirroring global id: d415bdfd-6a78-4ae5-a651-33437011bbb0
        mirroring primary: true


Slave:
rbd image 'test1':
        size 10240 MB in 2560 objects
        order 22 (4096 kB objects)
        block_name_prefix: rbd_data.10783837f1df
        format: 2
        features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling
        flags: 
        journal: 10783837f1df
        mirroring state: enabled
        mirroring global id: d415bdfd-6a78-4ae5-a651-33437011bbb0
        mirroring primary: false


Actual results:
I am seeing Object Map and fast diff getting invalid on the Master Node.

Expected results:
Flag should not get invalid.

Additional info:
Master Log
There were no crash.

-------------------------------------------------------------------------------

Master:

systemctl status -l ceph-rbd-mirror@master
● ceph-rbd-mirror@master.service - Ceph rbd mirror daemon
   Loaded: loaded (/usr/lib/systemd/system/ceph-rbd-mirror@.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2016-06-16 11:15:02 IST; 2h 52min ago
 Main PID: 66591 (rbd-mirror)
   CGroup: /system.slice/system-ceph\x2drbd\x2dmirror.slice/ceph-rbd-mirror@master.service
           └─66591 /usr/bin/rbd-mirror -f --cluster master --id master --setuser ceph --setgroup ceph

Jun 16 13:56:24 x.x rbd-mirror[66591]: 2016-06-16 13:56:24.254626 7f702f7fe700 -1 JournalMetadata: operator(): failed to watch journal(2) No such file or directory
Jun 16 13:56:24 x.x rbd-mirror[66591]: 2016-06-16 13:56:24.254646 7f702f7fe700 -1 JournalMetadata: failed to initialize immutable metadata: (2) No such file or directory
Jun 16 13:56:54 x.x rbd-mirror[66591]: 2016-06-16 13:56:54.245727 7f702f7fe700 -1 JournalMetadata: operator(): failed to watch journal(2) No such file or directory
Jun 16 13:56:54 x.x rbd-mirror[66591]: 2016-06-16 13:56:54.245738 7f702f7fe700 -1 JournalMetadata: failed to initialize immutable metadata: (2) No such file or directory
Jun 16 13:58:27 x.x rbd-mirror[66591]: 2016-06-16 13:58:27.540204 7f701cff9700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x7f6fe8004b30 handle_get_remote_tag_class: failed to retrieve remote client: (2) No such file or directory
Jun 16 13:58:27 x.x rbd-mirror[66591]: 2016-06-16 13:58:27.540278 7f70508ee700 -1 rbd::mirror::ImageReplayer: 0x7f6fe8001ea0 [1/659a83ea-3ead-4cc1-8b5b-8c1ffb4f327b] operator(): start failed: (2) No such file or directory
Jun 16 13:58:27 x.x rbd-mirror[66591]: 2016-06-16 13:58:27.552084 7f702f7fe700 -1 JournalMetadata: operator(): failed to watch journal(2) No such file or directory
Jun 16 13:58:27 x.x rbd-mirror[66591]: 2016-06-16 13:58:27.552103 7f702f7fe700 -1 JournalMetadata: failed to initialize immutable metadata: (2) No such file or directory
Jun 16 13:58:57 x.x rbd-mirror[66591]: 2016-06-16 13:58:57.529338 7f702f7fe700 -1 JournalMetadata: operator(): failed to watch journal(2) No such file or directory
Jun 16 13:58:57 x.x rbd-mirror[66591]: 2016-06-16 13:58:57.529359 7f702f7fe700 -1 JournalMetadata: failed to initialize immutable metadata: (2) No such file or directory




Slave:

systemctl status -l ceph-rbd-mirror@slave
● ceph-rbd-mirror@slave.service - Ceph rbd mirror daemon
   Loaded: loaded (/usr/lib/systemd/system/ceph-rbd-mirror@.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2016-06-15 16:53:29 UTC; 15h ago
 Main PID: 30993 (rbd-mirror)
   CGroup: /system.slice/system-ceph\x2drbd\x2dmirror.slice/ceph-rbd-mirror@slave.service
           └─30993 /usr/bin/rbd-mirror -f --cluster slave --id slave --setuser ceph --setgroup ceph

Jun 16 08:28:12 magna003 rbd-mirror[30993]: 2016-06-16 08:28:12.139102 7f8d41ffb700 -1 rbd::mirror::image_sync::ImageCopyRequest: 0x7f8cf000bab0 handle_object_copy: object copy failed: (2) No such file or directory
Jun 16 08:28:22 magna003 rbd-mirror[30993]: 2016-06-16 08:28:22.573271 7f8d41ffb700 -1 rbd::mirror::image_sync::ObjectCopyRequest: 0x7f8d04104100 handle_read_object: failed to read from remote object: (2) No such file or directory
Jun 16 08:28:22 magna003 rbd-mirror[30993]: 2016-06-16 08:28:22.573285 7f8d41ffb700 -1 rbd::mirror::image_sync::ImageCopyRequest: 0x7f8cf000bab0 handle_object_copy: object copy failed: (2) No such file or directory
Jun 16 08:28:22 magna003 rbd-mirror[30993]: 2016-06-16 08:28:22.573293 7f8d41ffb700 -1 rbd::mirror::ImageSync: 0x7f8cf0056210 handle_copy_image: failed to copy image: (2) No such file or directory
Jun 16 08:28:22 magna003 rbd-mirror[30993]: 2016-06-16 08:28:22.573297 7f8d41ffb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x7f8d0002a330 handle_image_sync: failed to sync remote image: (2) No such file or directory
Jun 16 08:28:22 magna003 rbd-mirror[30993]: 2016-06-16 08:28:22.883709 7f8d6981b700 -1 rbd::mirror::ImageReplayer: 0x7f8d000065c0 [1/659a83ea-3ead-4cc1-8b5b-8c1ffb4f327b] operator(): start failed: (2) No such file or directory
Jun 16 08:28:34 magna003 rbd-mirror[30993]: 2016-06-16 08:28:34.176266 7f8d41ffb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x7f8d00010e50 handle_get_remote_tag_class: failed to retrieve remote client: (2) No such file or directory
Jun 16 08:28:34 magna003 rbd-mirror[30993]: 2016-06-16 08:28:34.176330 7f8d6981b700 -1 rbd::mirror::ImageReplayer: 0x7f8d000065c0 [1/659a83ea-3ead-4cc1-8b5b-8c1ffb4f327b] operator(): start failed: (2) No such file or directory
Jun 16 08:29:04 magna003 rbd-mirror[30993]: 2016-06-16 08:29:04.175564 7f8d58ef8700 -1 JournalMetadata: operator(): failed to watch journal(2) No such file or directory
Jun 16 08:29:04 magna003 rbd-mirror[30993]: 2016-06-16 08:29:04.175584 7f8d58ef8700 -1 JournalMetadata: failed to initialize immutable metadata: (2) No such file or directory

Comment 2 Tanay Ganguly 2016-06-16 08:42:29 UTC
Created attachment 1168635 [details]
Slave Log

Comment 3 Tanay Ganguly 2016-06-16 09:06:51 UTC
Version:
ceph version 10.2.2-1.el7cp
rbd-mirror-10.2.2-1.el7cp.x86_64

Reproduced it Once.

Comment 10 Jason Dillaman 2016-08-10 19:16:52 UTC
Upstream pull request: https://github.com/ceph/ceph/pull/9899

Comment 11 kiran raje urs J 2016-09-14 09:39:17 UTC
changing the need info from Tanay to Hemanth.

Comment 16 Rachana Patel 2016-10-21 19:34:50 UTC
verified with 10.2.3-8.el7cp.x86_64.

Followed procedure mentioned in bug description. no error messages hence moving to verified

Comment 18 errata-xmlrpc 2016-11-22 19:26:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2815.html


Note You need to log in before you can comment on or make changes to this bug.