Bug 1347174

Summary:

Object Map and Fast Diff Flag is getting invalid, while doing multiple times enable/disable of Journaling on an Image

Product:

[Red Hat Storage] Red Hat Ceph Storage

Reporter:

Tanay Ganguly <tganguly>

Component:

RBD

Assignee:

Jason Dillaman <jdillama>

Status:

CLOSED ERRATA

QA Contact:

Rachana Patel <racpatel>

Severity:

high

Docs Contact:

Priority:

medium

Version:

2.0

CC:

ceph-eng-bugs, flucifre, hnallurv, hyelloji, kdreyer, kurs, uboppana

Target Milestone:

Flags:

hyelloji: needinfo+

Target Release:

2.1

Hardware:

x86_64

OS:

Linux

Whiteboard:

Fixed In Version:

RHEL: ceph-10.2.3-2.el7cp Ubuntu: ceph_10.2.3-3redhat1xenial

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2016-11-22 19:26:33 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
Master Log	none
Slave Log	none

Description Tanay Ganguly 2016-06-16 08:41:34 UTC

Created attachment 1168632 [details]
Master Log

Description of problem:
object map invalid, fast diff invalid

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Create an Image with Journaling enabled.
2. Start IO using bench-write.
3. Then execute:

for i in {1..10}; do rbd feature disable RBD/test1 journaling --cluster master; rbd bench-write RBD/test1 --io-size 10240 --cluster master; rbd feature enable RBD/test1 journaling --cluster master; done

Master:
rbd image 'test1':
        size 10240 MB in 2560 objects
        order 22 (4096 kB objects)
        block_name_prefix: rbd_data.11052ae8944a
        format: 2
        features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling
        flags: object map invalid, fast diff invalid
        journal: 11052ae8944a
        mirroring state: enabled
        mirroring global id: d415bdfd-6a78-4ae5-a651-33437011bbb0
        mirroring primary: true


Slave:
rbd image 'test1':
        size 10240 MB in 2560 objects
        order 22 (4096 kB objects)
        block_name_prefix: rbd_data.10783837f1df
        format: 2
        features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling
        flags: 
        journal: 10783837f1df
        mirroring state: enabled
        mirroring global id: d415bdfd-6a78-4ae5-a651-33437011bbb0
        mirroring primary: false


Actual results:
I am seeing Object Map and fast diff getting invalid on the Master Node.

Expected results:
Flag should not get invalid.

Additional info:
Master Log
There were no crash.

-------------------------------------------------------------------------------

Master:

systemctl status -l ceph-rbd-mirror@master
● ceph-rbd-mirror - Ceph rbd mirror daemon
   Loaded: loaded (/usr/lib/systemd/system/ceph-rbd-mirror@.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2016-06-16 11:15:02 IST; 2h 52min ago
 Main PID: 66591 (rbd-mirror)
   CGroup: /system.slice/system-ceph\x2drbd\x2dmirror.slice/ceph-rbd-mirror
           └─66591 /usr/bin/rbd-mirror -f --cluster master --id master --setuser ceph --setgroup ceph

Jun 16 13:56:24 x.x rbd-mirror[66591]: 2016-06-16 13:56:24.254626 7f702f7fe700 -1 JournalMetadata: operator(): failed to watch journal(2) No such file or directory
Jun 16 13:56:24 x.x rbd-mirror[66591]: 2016-06-16 13:56:24.254646 7f702f7fe700 -1 JournalMetadata: failed to initialize immutable metadata: (2) No such file or directory
Jun 16 13:56:54 x.x rbd-mirror[66591]: 2016-06-16 13:56:54.245727 7f702f7fe700 -1 JournalMetadata: operator(): failed to watch journal(2) No such file or directory
Jun 16 13:56:54 x.x rbd-mirror[66591]: 2016-06-16 13:56:54.245738 7f702f7fe700 -1 JournalMetadata: failed to initialize immutable metadata: (2) No such file or directory
Jun 16 13:58:27 x.x rbd-mirror[66591]: 2016-06-16 13:58:27.540204 7f701cff9700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x7f6fe8004b30 handle_get_remote_tag_class: failed to retrieve remote client: (2) No such file or directory
Jun 16 13:58:27 x.x rbd-mirror[66591]: 2016-06-16 13:58:27.540278 7f70508ee700 -1 rbd::mirror::ImageReplayer: 0x7f6fe8001ea0 [1/659a83ea-3ead-4cc1-8b5b-8c1ffb4f327b] operator(): start failed: (2) No such file or directory
Jun 16 13:58:27 x.x rbd-mirror[66591]: 2016-06-16 13:58:27.552084 7f702f7fe700 -1 JournalMetadata: operator(): failed to watch journal(2) No such file or directory
Jun 16 13:58:27 x.x rbd-mirror[66591]: 2016-06-16 13:58:27.552103 7f702f7fe700 -1 JournalMetadata: failed to initialize immutable metadata: (2) No such file or directory
Jun 16 13:58:57 x.x rbd-mirror[66591]: 2016-06-16 13:58:57.529338 7f702f7fe700 -1 JournalMetadata: operator(): failed to watch journal(2) No such file or directory
Jun 16 13:58:57 x.x rbd-mirror[66591]: 2016-06-16 13:58:57.529359 7f702f7fe700 -1 JournalMetadata: failed to initialize immutable metadata: (2) No such file or directory




Slave:

systemctl status -l ceph-rbd-mirror@slave
● ceph-rbd-mirror - Ceph rbd mirror daemon
   Loaded: loaded (/usr/lib/systemd/system/ceph-rbd-mirror@.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2016-06-15 16:53:29 UTC; 15h ago
 Main PID: 30993 (rbd-mirror)
   CGroup: /system.slice/system-ceph\x2drbd\x2dmirror.slice/ceph-rbd-mirror
           └─30993 /usr/bin/rbd-mirror -f --cluster slave --id slave --setuser ceph --setgroup ceph

Jun 16 08:28:12 magna003 rbd-mirror[30993]: 2016-06-16 08:28:12.139102 7f8d41ffb700 -1 rbd::mirror::image_sync::ImageCopyRequest: 0x7f8cf000bab0 handle_object_copy: object copy failed: (2) No such file or directory
Jun 16 08:28:22 magna003 rbd-mirror[30993]: 2016-06-16 08:28:22.573271 7f8d41ffb700 -1 rbd::mirror::image_sync::ObjectCopyRequest: 0x7f8d04104100 handle_read_object: failed to read from remote object: (2) No such file or directory
Jun 16 08:28:22 magna003 rbd-mirror[30993]: 2016-06-16 08:28:22.573285 7f8d41ffb700 -1 rbd::mirror::image_sync::ImageCopyRequest: 0x7f8cf000bab0 handle_object_copy: object copy failed: (2) No such file or directory
Jun 16 08:28:22 magna003 rbd-mirror[30993]: 2016-06-16 08:28:22.573293 7f8d41ffb700 -1 rbd::mirror::ImageSync: 0x7f8cf0056210 handle_copy_image: failed to copy image: (2) No such file or directory
Jun 16 08:28:22 magna003 rbd-mirror[30993]: 2016-06-16 08:28:22.573297 7f8d41ffb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x7f8d0002a330 handle_image_sync: failed to sync remote image: (2) No such file or directory
Jun 16 08:28:22 magna003 rbd-mirror[30993]: 2016-06-16 08:28:22.883709 7f8d6981b700 -1 rbd::mirror::ImageReplayer: 0x7f8d000065c0 [1/659a83ea-3ead-4cc1-8b5b-8c1ffb4f327b] operator(): start failed: (2) No such file or directory
Jun 16 08:28:34 magna003 rbd-mirror[30993]: 2016-06-16 08:28:34.176266 7f8d41ffb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x7f8d00010e50 handle_get_remote_tag_class: failed to retrieve remote client: (2) No such file or directory
Jun 16 08:28:34 magna003 rbd-mirror[30993]: 2016-06-16 08:28:34.176330 7f8d6981b700 -1 rbd::mirror::ImageReplayer: 0x7f8d000065c0 [1/659a83ea-3ead-4cc1-8b5b-8c1ffb4f327b] operator(): start failed: (2) No such file or directory
Jun 16 08:29:04 magna003 rbd-mirror[30993]: 2016-06-16 08:29:04.175564 7f8d58ef8700 -1 JournalMetadata: operator(): failed to watch journal(2) No such file or directory
Jun 16 08:29:04 magna003 rbd-mirror[30993]: 2016-06-16 08:29:04.175584 7f8d58ef8700 -1 JournalMetadata: failed to initialize immutable metadata: (2) No such file or directory

Comment 2 Tanay Ganguly 2016-06-16 08:42:29 UTC

Created attachment 1168635 [details]
Slave Log

Comment 3 Tanay Ganguly 2016-06-16 09:06:51 UTC

Version:
ceph version 10.2.2-1.el7cp
rbd-mirror-10.2.2-1.el7cp.x86_64

Reproduced it Once.

Comment 10 Jason Dillaman 2016-08-10 19:16:52 UTC

Upstream pull request: https://github.com/ceph/ceph/pull/9899

Comment 11 kiran raje urs J 2016-09-14 09:39:17 UTC

changing the need info from Tanay to Hemanth.

Comment 16 Rachana Patel 2016-10-21 19:34:50 UTC

verified with 10.2.3-8.el7cp.x86_64.

Followed procedure mentioned in bug description. no error messages hence moving to verified

Comment 18 errata-xmlrpc 2016-11-22 19:26:33 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2815.html