Bug 1327412 - [RBD] failed to update object map during rbd flatten
Summary: [RBD] failed to update object map during rbd flatten
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RBD
Version: 2.0
Hardware: Unspecified
OS: Linux
unspecified
medium
Target Milestone: rc
: 2.0
Assignee: Jason Dillaman
QA Contact: Hemanth Kumar
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-04-15 06:28 UTC by Tejas
Modified: 2017-07-30 15:30 UTC (History)
6 users (show)

Fixed In Version: RHEL: ceph-10.2.1-1.el7cp Ubuntu: ceph_10.2.1-2redhat1xenial
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-08-23 19:36:29 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Ceph Project Bug Tracker 15572 None None None 2016-04-22 14:06:52 UTC
Ceph Project Bug Tracker 16114 None None None 2016-06-03 10:59:50 UTC
Red Hat Product Errata RHBA-2016:1755 normal SHIPPED_LIVE Red Hat Ceph Storage 2.0 bug fix and enhancement update 2016-08-23 23:23:52 UTC

Description Tejas 2016-04-15 06:28:09 UTC
Description of problem:
If object map feature is disabled on a clone while flatten is in progress, we see a spew of logs "failed to update object map".

Version-Release number of selected component (if applicable):
ceph 10.1.1

How reproducible:
Always

Steps to Reproduce:
1. Create an image, write some data on it.
2. Take a snap,protect and clone the snap.
3. Write some data on the snap, create a snap on it and  start flatten.
4. While flatten is in progress, disable the object-map on the clone.

Actual results:
seeing lots of messages such as "failed to update object map: (2) No such file or directory"

Expected results:
Object-map invalidation should not fail. And error messages needed to be logged appropriately

Additional info:

[root@magna009 ~]# rbd flatten Tejas/c1
Image flatten: 6% complete...2016-04-15 06:13:11.817643 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.817677 7f89ba8df700 -1 librbd::object_map::InvalidateRequest: 0x7f89a0051dc0 invalidating object map in-memory
2016-04-15 06:13:11.817679 7f89ba8df700 -1 librbd::object_map::InvalidateRequest: 0x7f89a0051dc0 invalidating object map on-disk
2016-04-15 06:13:11.817729 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.817766 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a0036ce0 should_complete: r=0
2016-04-15 06:13:11.817806 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.817885 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.818022 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.818121 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.818257 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a0025350 should_complete: r=0
2016-04-15 06:13:11.818720 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a001cba0 should_complete: r=0
2016-04-15 06:13:11.819164 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a0061ec0 should_complete: r=0
2016-04-15 06:13:11.819614 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a0033030 should_complete: r=0
2016-04-15 06:13:11.832618 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.832669 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a002c8a0 should_complete: r=0
2016-04-15 06:13:11.849946 7f89ba8df700 -1 librbd::object_map::InvalidateRequest: 0x7f89a0051dc0 should_complete: r=0
2016-04-15 06:13:11.874390 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.874439 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a0020b80 should_complete: r=0
2016-04-15 06:13:11.891320 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.891371 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a0048370 should_complete: r=0
2016-04-15 06:13:11.899605 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.899658 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a00843d0 should_complete: r=0
2016-04-15 06:13:11.908182 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.908233 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a00843d0 should_complete: r=0
2016-04-15 06:13:11.921060 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.921108 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a006c760 should_complete: r=0
2016-04-15 06:13:11.929212 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.929235 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a006c760 should_complete: r=0
2016-04-15 06:13:11.944807 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.944826 7f89ba8df700 -1 librbd::object_map::InvalidateRequest: 0x7f89a00235c0 invalidating object map in-memory
2016-04-15 06:13:11.944829 7f89ba8df700 -1 librbd::object_map::InvalidateRequest: 0x7f89a00235c0 invalidating object map on-disk
2016-04-15 06:13:11.944870 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.944942 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.945001 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.945082 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.945201 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.945300 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.945409 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.945534 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.945716 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
.
.
.
2016-04-15 06:14:05.361410 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:14:05.362501 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:14:05.363357 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:14:05.364202 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:14:05.365041 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:14:05.365882 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:14:05.366706 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:14:05.367127 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
Image flatten: 100% complete...done.
[root@magna009 ~]# 
[root@magna009 ~]# 
[root@magna009 ~]# 
[root@magna009 ~]# rbd ls -l Tejas
NAME    SIZE PARENT FMT PROT LOCK 
b1    15360M          2           
b1@s1 15360M          2 yes       
c1    15360M          2           
c1@s2 15360M          2           
[root@magna009 ~]#

Comment 2 Jason Dillaman 2016-04-28 19:11:02 UTC
Upstream PR: https://github.com/ceph/ceph/pull/8817

Comment 3 Ken Dreyer (Red Hat) 2016-05-10 13:22:12 UTC
This is undergoing review upstream (https://github.com/ceph/ceph/pull/8869) and will be in v10.2.1.

Comment 4 Ken Dreyer (Red Hat) 2016-05-16 15:30:57 UTC
The above PR was merged to jewel and is present in v10.2.1.

Comment 7 Hemanth Kumar 2016-05-31 12:09:21 UTC
I am unable to disable the object-map while the flattening is in-progress

# rbd flatten pool4/bucket11 --cluster master
Image flatten: 69% complete...done.

# rbd feature disable pool4/bucket11 object-map --cluster master
rbd: failed to update image features: (22) Invalid argument2016-05-31 12:01:39.088013 7f5d1fa8fd80 -1 librbd: cannot disable object map

# rbd ls -l bucket10 -p pool4 --cluster master 
NAME             SIZE PARENT FMT PROT LOCK 
bucket10       10240M          2           
bucket10@shot1 10240M          2 yes       
bucket11       10240M          2           
bucket11@shot2 10240M          2           

===========================================================================

Comment 8 Jason Dillaman 2016-05-31 15:26:36 UTC
@Hamanth: did the image also have fast-diff enabled?

Comment 9 Hemanth Kumar 2016-06-02 10:46:14 UTC
(In reply to Jason Dillaman from comment #8)
> @Hamanth: did the image also have fast-diff enabled?

Yes Jason. fast-diff was also enabled.
BTW, all the features gets enabled by default with the latest ceph.

Comment 10 Jason Dillaman 2016-06-02 12:46:26 UTC
@Hemanth: moving this back to ON_QA as it isn't a bug.  Since fast-diff depends on object-map, you cannot disable object-map without also disabling fast-diff.

Comment 11 Hemanth Kumar 2016-06-03 08:09:23 UTC
(In reply to Jason Dillaman from comment #10)
> @Hemanth: moving this back to ON_QA as it isn't a bug.  Since fast-diff
> depends on object-map, you cannot disable object-map without also disabling
> fast-diff.


@Jason.,
Error messages are still seen while disabling the object map during flattening.


# rbd flatten rbd/image2
Image flatten: 26% complete...2016-06-03 08:01:28.718143 7f3c2ffff700 -1 librbd::object_map::RefreshRequest: failed to load object map: rbd_object_map.17ec5c238e1f29
2016-06-03 08:01:28.718310 7f3c2ffff700 -1 librbd::object_map::InvalidateRequest: 0x7f3c14053500 invalidating object map in-memory
2016-06-03 08:01:28.718322 7f3c2f7fe700 -1 librbd::object_map::InvalidateRequest: 0x7f3c14053500 should_complete: r=0
Image flatten: 100% complete...done.

Disabled the object-map during flattening..
-----------------------------------------
# rbd info image2
rbd image 'image2':
	size 10240 MB in 2560 objects
	order 22 (4096 kB objects)
	block_name_prefix: rbd_data.17ec5c238e1f29
	format: 2
	features: layering, exclusive-lock, object-map, deep-flatten
# rbd feature disable rbd/image2  object-map 
# rbd info image2
rbd image 'image2':
	size 10240 MB in 2560 objects
	order 22 (4096 kB objects)
	block_name_prefix: rbd_data.17ec5c238e1f29
	format: 2
	features: layering, exclusive-lock, deep-flatten
	flags: 

Ceph version :- 10.2.1-7.el7cp.x86_64

Comment 12 Jason Dillaman 2016-06-03 11:26:44 UTC
Same issue as disabling journal during in-progress operation.  Upstream PR: https://github.com/ceph/ceph/pull/9440

Comment 15 Jason Dillaman 2016-06-10 17:37:09 UTC
Merged Jewel PR: https://github.com/ceph/ceph/pull/9610

Comment 16 Ken Dreyer (Red Hat) 2016-06-14 16:24:13 UTC
This is in v10.2.2 upstream, so we will take this change in as part of the rebase to 10.2.2.

Comment 20 errata-xmlrpc 2016-08-23 19:36:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1755.html


Note You need to log in before you can comment on or make changes to this bug.