Description of problem: If object map feature is disabled on a clone while flatten is in progress, we see a spew of logs "failed to update object map". Version-Release number of selected component (if applicable): ceph 10.1.1 How reproducible: Always Steps to Reproduce: 1. Create an image, write some data on it. 2. Take a snap,protect and clone the snap. 3. Write some data on the snap, create a snap on it and start flatten. 4. While flatten is in progress, disable the object-map on the clone. Actual results: seeing lots of messages such as "failed to update object map: (2) No such file or directory" Expected results: Object-map invalidation should not fail. And error messages needed to be logged appropriately Additional info: [root@magna009 ~]# rbd flatten Tejas/c1 Image flatten: 6% complete...2016-04-15 06:13:11.817643 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2016-04-15 06:13:11.817677 7f89ba8df700 -1 librbd::object_map::InvalidateRequest: 0x7f89a0051dc0 invalidating object map in-memory 2016-04-15 06:13:11.817679 7f89ba8df700 -1 librbd::object_map::InvalidateRequest: 0x7f89a0051dc0 invalidating object map on-disk 2016-04-15 06:13:11.817729 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2016-04-15 06:13:11.817766 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a0036ce0 should_complete: r=0 2016-04-15 06:13:11.817806 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2016-04-15 06:13:11.817885 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2016-04-15 06:13:11.818022 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2016-04-15 06:13:11.818121 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2016-04-15 06:13:11.818257 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a0025350 should_complete: r=0 2016-04-15 06:13:11.818720 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a001cba0 should_complete: r=0 2016-04-15 06:13:11.819164 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a0061ec0 should_complete: r=0 2016-04-15 06:13:11.819614 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a0033030 should_complete: r=0 2016-04-15 06:13:11.832618 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2016-04-15 06:13:11.832669 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a002c8a0 should_complete: r=0 2016-04-15 06:13:11.849946 7f89ba8df700 -1 librbd::object_map::InvalidateRequest: 0x7f89a0051dc0 should_complete: r=0 2016-04-15 06:13:11.874390 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2016-04-15 06:13:11.874439 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a0020b80 should_complete: r=0 2016-04-15 06:13:11.891320 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2016-04-15 06:13:11.891371 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a0048370 should_complete: r=0 2016-04-15 06:13:11.899605 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2016-04-15 06:13:11.899658 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a00843d0 should_complete: r=0 2016-04-15 06:13:11.908182 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2016-04-15 06:13:11.908233 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a00843d0 should_complete: r=0 2016-04-15 06:13:11.921060 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2016-04-15 06:13:11.921108 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a006c760 should_complete: r=0 2016-04-15 06:13:11.929212 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2016-04-15 06:13:11.929235 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a006c760 should_complete: r=0 2016-04-15 06:13:11.944807 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2016-04-15 06:13:11.944826 7f89ba8df700 -1 librbd::object_map::InvalidateRequest: 0x7f89a00235c0 invalidating object map in-memory 2016-04-15 06:13:11.944829 7f89ba8df700 -1 librbd::object_map::InvalidateRequest: 0x7f89a00235c0 invalidating object map on-disk 2016-04-15 06:13:11.944870 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2016-04-15 06:13:11.944942 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2016-04-15 06:13:11.945001 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2016-04-15 06:13:11.945082 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2016-04-15 06:13:11.945201 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2016-04-15 06:13:11.945300 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2016-04-15 06:13:11.945409 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2016-04-15 06:13:11.945534 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2016-04-15 06:13:11.945716 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory . . . 2016-04-15 06:14:05.361410 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2016-04-15 06:14:05.362501 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2016-04-15 06:14:05.363357 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2016-04-15 06:14:05.364202 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2016-04-15 06:14:05.365041 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2016-04-15 06:14:05.365882 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2016-04-15 06:14:05.366706 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory 2016-04-15 06:14:05.367127 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory Image flatten: 100% complete...done. [root@magna009 ~]# [root@magna009 ~]# [root@magna009 ~]# [root@magna009 ~]# rbd ls -l Tejas NAME SIZE PARENT FMT PROT LOCK b1 15360M 2 b1@s1 15360M 2 yes c1 15360M 2 c1@s2 15360M 2 [root@magna009 ~]#
Upstream PR: https://github.com/ceph/ceph/pull/8817
This is undergoing review upstream (https://github.com/ceph/ceph/pull/8869) and will be in v10.2.1.
The above PR was merged to jewel and is present in v10.2.1.
I am unable to disable the object-map while the flattening is in-progress # rbd flatten pool4/bucket11 --cluster master Image flatten: 69% complete...done. # rbd feature disable pool4/bucket11 object-map --cluster master rbd: failed to update image features: (22) Invalid argument2016-05-31 12:01:39.088013 7f5d1fa8fd80 -1 librbd: cannot disable object map # rbd ls -l bucket10 -p pool4 --cluster master NAME SIZE PARENT FMT PROT LOCK bucket10 10240M 2 bucket10@shot1 10240M 2 yes bucket11 10240M 2 bucket11@shot2 10240M 2 ===========================================================================
@Hamanth: did the image also have fast-diff enabled?
(In reply to Jason Dillaman from comment #8) > @Hamanth: did the image also have fast-diff enabled? Yes Jason. fast-diff was also enabled. BTW, all the features gets enabled by default with the latest ceph.
@Hemanth: moving this back to ON_QA as it isn't a bug. Since fast-diff depends on object-map, you cannot disable object-map without also disabling fast-diff.
(In reply to Jason Dillaman from comment #10) > @Hemanth: moving this back to ON_QA as it isn't a bug. Since fast-diff > depends on object-map, you cannot disable object-map without also disabling > fast-diff. @Jason., Error messages are still seen while disabling the object map during flattening. # rbd flatten rbd/image2 Image flatten: 26% complete...2016-06-03 08:01:28.718143 7f3c2ffff700 -1 librbd::object_map::RefreshRequest: failed to load object map: rbd_object_map.17ec5c238e1f29 2016-06-03 08:01:28.718310 7f3c2ffff700 -1 librbd::object_map::InvalidateRequest: 0x7f3c14053500 invalidating object map in-memory 2016-06-03 08:01:28.718322 7f3c2f7fe700 -1 librbd::object_map::InvalidateRequest: 0x7f3c14053500 should_complete: r=0 Image flatten: 100% complete...done. Disabled the object-map during flattening.. ----------------------------------------- # rbd info image2 rbd image 'image2': size 10240 MB in 2560 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.17ec5c238e1f29 format: 2 features: layering, exclusive-lock, object-map, deep-flatten # rbd feature disable rbd/image2 object-map # rbd info image2 rbd image 'image2': size 10240 MB in 2560 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.17ec5c238e1f29 format: 2 features: layering, exclusive-lock, deep-flatten flags: Ceph version :- 10.2.1-7.el7cp.x86_64
Same issue as disabling journal during in-progress operation. Upstream PR: https://github.com/ceph/ceph/pull/9440
Merged Jewel PR: https://github.com/ceph/ceph/pull/9610
This is in v10.2.2 upstream, so we will take this change in as part of the rebase to 10.2.2.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-1755.html