Bug 1347195

Summary: Unable to Enable Journaling on an Image
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Tanay Ganguly <tganguly>
Component: RBDAssignee: Jason Dillaman <jdillama>
Status: CLOSED ERRATA QA Contact: Tanay Ganguly <tganguly>
Severity: high Docs Contact:
Priority: unspecified    
Version: 2.0CC: ceph-eng-bugs, hnallurv, kdreyer, kurs, tganguly
Target Milestone: rc   
Target Release: 2.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: RHEL: ceph-10.2.2-3.el7cp Ubuntu: ceph_10.2.2-4redhat1xenial Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-08-23 19:41:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1343229    

Description Tanay Ganguly 2016-06-16 09:34:06 UTC 
Description of problem:
Even after enabling Journal, i am seeing the Mirroring state as Disabled and the image is not getting synced to Slave Node.

Version-Release number of selected component (if applicable):
ceph version 10.2.2-1.el7cp
rbd-mirror-10.2.2-1.el7cp.x86_64

How reproducible:
Once

Steps to Reproduce:
While Reproducing BZ 1347174, same steps as the mentioned BUG

On one of my Image test2, mirroring state is showing as disabled.
Even after enabling the Journaling on the Image, its unable to sync with the Slave Node. Although Journaling feature is showing as enabled but the functionality is not working.

Master:
rbd image 'test2':
        size 10240 MB in 2560 objects
        order 22 (4096 kB objects)
        block_name_prefix: rbd_data.1111238e1f29
        format: 2
        features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling
        flags: 
        journal: 1111238e1f29
        mirroring state: disabled

Slave:
rbd: error opening image test22016-06-16 09:09:42.499601 7ff16cff9700 10 librbd::image::OpenRequest: handle_close_image: r=0
: (2) No such file or directory
2016-06-16 09:09:42.499609 7ff16cff9700 10 librbd::ImageState: 0x7ff19b28f070 handle_open: r=-2


Actual results:
Although Journal is showing enabled, but its not working as per expected. Its not syncing to Slave Node.

Expected results:
Journaling should get enabled and the image should sync.

Additional info:


----------------------------------------------------------------------------
Master:

systemctl status -l ceph-rbd-mirror@master
● ceph-rbd-mirror - Ceph rbd mirror daemon
   Loaded: loaded (/usr/lib/systemd/system/ceph-rbd-mirror@.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2016-06-16 11:15:02 IST; 2h 52min ago
 Main PID: 66591 (rbd-mirror)
   CGroup: /system.slice/system-ceph\x2drbd\x2dmirror.slice/ceph-rbd-mirror
           └─66591 /usr/bin/rbd-mirror -f --cluster master --id master --setuser ceph --setgroup ceph

Jun 16 13:56:24 x.x rbd-mirror[66591]: 2016-06-16 13:56:24.254626 7f702f7fe700 -1 JournalMetadata: operator(): failed to watch journal(2) No such file or directory
Jun 16 13:56:24 x.x rbd-mirror[66591]: 2016-06-16 13:56:24.254646 7f702f7fe700 -1 JournalMetadata: failed to initialize immutable metadata: (2) No such file or directory
Jun 16 13:56:54 x.x rbd-mirror[66591]: 2016-06-16 13:56:54.245727 7f702f7fe700 -1 JournalMetadata: operator(): failed to watch journal(2) No such file or directory
Jun 16 13:56:54 x.x rbd-mirror[66591]: 2016-06-16 13:56:54.245738 7f702f7fe700 -1 JournalMetadata: failed to initialize immutable metadata: (2) No such file or directory
Jun 16 13:58:27 x.x rbd-mirror[66591]: 2016-06-16 13:58:27.540204 7f701cff9700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x7f6fe8004b30 handle_get_remote_tag_class: failed to retrieve remote client: (2) No such file or directory
Jun 16 13:58:27 x.x rbd-mirror[66591]: 2016-06-16 13:58:27.540278 7f70508ee700 -1 rbd::mirror::ImageReplayer: 0x7f6fe8001ea0 [1/659a83ea-3ead-4cc1-8b5b-8c1ffb4f327b] operator(): start failed: (2) No such file or directory
Jun 16 13:58:27 x.x rbd-mirror[66591]: 2016-06-16 13:58:27.552084 7f702f7fe700 -1 JournalMetadata: operator(): failed to watch journal(2) No such file or directory
Jun 16 13:58:27 x.x rbd-mirror[66591]: 2016-06-16 13:58:27.552103 7f702f7fe700 -1 JournalMetadata: failed to initialize immutable metadata: (2) No such file or directory
Jun 16 13:58:57 x.x rbd-mirror[66591]: 2016-06-16 13:58:57.529338 7f702f7fe700 -1 JournalMetadata: operator(): failed to watch journal(2) No such file or directory
Jun 16 13:58:57 x.x rbd-mirror[66591]: 2016-06-16 13:58:57.529359 7f702f7fe700 -1 JournalMetadata: failed to initialize immutable metadata: (2) No such file or directory




Slave:

systemctl status -l ceph-rbd-mirror@slave
● ceph-rbd-mirror - Ceph rbd mirror daemon
   Loaded: loaded (/usr/lib/systemd/system/ceph-rbd-mirror@.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2016-06-15 16:53:29 UTC; 15h ago
 Main PID: 30993 (rbd-mirror)
   CGroup: /system.slice/system-ceph\x2drbd\x2dmirror.slice/ceph-rbd-mirror
           └─30993 /usr/bin/rbd-mirror -f --cluster slave --id slave --setuser ceph --setgroup ceph

Jun 16 08:28:12 magna003 rbd-mirror[30993]: 2016-06-16 08:28:12.139102 7f8d41ffb700 -1 rbd::mirror::image_sync::ImageCopyRequest: 0x7f8cf000bab0 handle_object_copy: object copy failed: (2) No such file or directory
Jun 16 08:28:22 magna003 rbd-mirror[30993]: 2016-06-16 08:28:22.573271 7f8d41ffb700 -1 rbd::mirror::image_sync::ObjectCopyRequest: 0x7f8d04104100 handle_read_object: failed to read from remote object: (2) No such file or directory
Jun 16 08:28:22 magna003 rbd-mirror[30993]: 2016-06-16 08:28:22.573285 7f8d41ffb700 -1 rbd::mirror::image_sync::ImageCopyRequest: 0x7f8cf000bab0 handle_object_copy: object copy failed: (2) No such file or directory
Jun 16 08:28:22 magna003 rbd-mirror[30993]: 2016-06-16 08:28:22.573293 7f8d41ffb700 -1 rbd::mirror::ImageSync: 0x7f8cf0056210 handle_copy_image: failed to copy image: (2) No such file or directory
Jun 16 08:28:22 magna003 rbd-mirror[30993]: 2016-06-16 08:28:22.573297 7f8d41ffb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x7f8d0002a330 handle_image_sync: failed to sync remote image: (2) No such file or directory
Jun 16 08:28:22 magna003 rbd-mirror[30993]: 2016-06-16 08:28:22.883709 7f8d6981b700 -1 rbd::mirror::ImageReplayer: 0x7f8d000065c0 [1/659a83ea-3ead-4cc1-8b5b-8c1ffb4f327b] operator(): start failed: (2) No such file or directory
Jun 16 08:28:34 magna003 rbd-mirror[30993]: 2016-06-16 08:28:34.176266 7f8d41ffb700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x7f8d00010e50 handle_get_remote_tag_class: failed to retrieve remote client: (2) No such file or directory
Jun 16 08:28:34 magna003 rbd-mirror[30993]: 2016-06-16 08:28:34.176330 7f8d6981b700 -1 rbd::mirror::ImageReplayer: 0x7f8d000065c0 [1/659a83ea-3ead-4cc1-8b5b-8c1ffb4f327b] operator(): start failed: (2) No such file or directory
Jun 16 08:29:04 magna003 rbd-mirror[30993]: 2016-06-16 08:29:04.175564 7f8d58ef8700 -1 JournalMetadata: operator(): failed to watch journal(2) No such file or directory
Jun 16 08:29:04 magna003 rbd-mirror[30993]: 2016-06-16 08:29:04.175584 7f8d58ef8700 -1 JournalMetadata: failed to initialize immutable metadata: (2) No such file or directory
Comment 8 Jason Dillaman 2016-06-16 17:13:13 UTC 
Upstream master branch PR: https://github.com/ceph/ceph/pull/9751
Comment 11 Tanay Ganguly 2016-07-07 07:41:07 UTC 
Unable to reproduce this Issue.

Marking it as Verified.
ceph version 10.2.2-15.el7cp
Comment 13 errata-xmlrpc 2016-08-23 19:41:52 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1755.html