Bug 1327412

Summary: [RBD] failed to update object map during rbd flatten
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Tejas <tchandra>
Component: RBDAssignee: Jason Dillaman <jdillama>
Status: CLOSED ERRATA QA Contact: Hemanth Kumar <hyelloji>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 2.0CC: ceph-eng-bugs, gmeno, hnallurv, hyelloji, jdillama, kdreyer
Target Milestone: rc   
Target Release: 2.0   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: RHEL: ceph-10.2.1-1.el7cp Ubuntu: ceph_10.2.1-2redhat1xenial Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-08-23 19:36:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Tejas 2016-04-15 06:28:09 UTC 
Description of problem:
If object map feature is disabled on a clone while flatten is in progress, we see a spew of logs "failed to update object map".

Version-Release number of selected component (if applicable):
ceph 10.1.1

How reproducible:
Always

Steps to Reproduce:
1. Create an image, write some data on it.
2. Take a snap,protect and clone the snap.
3. Write some data on the snap, create a snap on it and  start flatten.
4. While flatten is in progress, disable the object-map on the clone.

Actual results:
seeing lots of messages such as "failed to update object map: (2) No such file or directory"

Expected results:
Object-map invalidation should not fail. And error messages needed to be logged appropriately

Additional info:

[root@magna009 ~]# rbd flatten Tejas/c1
Image flatten: 6% complete...2016-04-15 06:13:11.817643 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.817677 7f89ba8df700 -1 librbd::object_map::InvalidateRequest: 0x7f89a0051dc0 invalidating object map in-memory
2016-04-15 06:13:11.817679 7f89ba8df700 -1 librbd::object_map::InvalidateRequest: 0x7f89a0051dc0 invalidating object map on-disk
2016-04-15 06:13:11.817729 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.817766 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a0036ce0 should_complete: r=0
2016-04-15 06:13:11.817806 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.817885 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.818022 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.818121 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.818257 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a0025350 should_complete: r=0
2016-04-15 06:13:11.818720 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a001cba0 should_complete: r=0
2016-04-15 06:13:11.819164 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a0061ec0 should_complete: r=0
2016-04-15 06:13:11.819614 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a0033030 should_complete: r=0
2016-04-15 06:13:11.832618 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.832669 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a002c8a0 should_complete: r=0
2016-04-15 06:13:11.849946 7f89ba8df700 -1 librbd::object_map::InvalidateRequest: 0x7f89a0051dc0 should_complete: r=0
2016-04-15 06:13:11.874390 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.874439 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a0020b80 should_complete: r=0
2016-04-15 06:13:11.891320 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.891371 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a0048370 should_complete: r=0
2016-04-15 06:13:11.899605 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.899658 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a00843d0 should_complete: r=0
2016-04-15 06:13:11.908182 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.908233 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a00843d0 should_complete: r=0
2016-04-15 06:13:11.921060 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.921108 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a006c760 should_complete: r=0
2016-04-15 06:13:11.929212 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.929235 7f89ba0de700 -1 librbd::object_map::InvalidateRequest: 0x7f89a006c760 should_complete: r=0
2016-04-15 06:13:11.944807 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.944826 7f89ba8df700 -1 librbd::object_map::InvalidateRequest: 0x7f89a00235c0 invalidating object map in-memory
2016-04-15 06:13:11.944829 7f89ba8df700 -1 librbd::object_map::InvalidateRequest: 0x7f89a00235c0 invalidating object map on-disk
2016-04-15 06:13:11.944870 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.944942 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.945001 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.945082 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.945201 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.945300 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.945409 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.945534 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:13:11.945716 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
.
.
.
2016-04-15 06:14:05.361410 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:14:05.362501 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:14:05.363357 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:14:05.364202 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:14:05.365041 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:14:05.365882 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:14:05.366706 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
2016-04-15 06:14:05.367127 7f89ba8df700 -1 librbd::object_map::Request: failed to update object map: (2) No such file or directory
Image flatten: 100% complete...done.
[root@magna009 ~]# 
[root@magna009 ~]# 
[root@magna009 ~]# 
[root@magna009 ~]# rbd ls -l Tejas
NAME    SIZE PARENT FMT PROT LOCK 
b1    15360M          2           
b1@s1 15360M          2 yes       
c1    15360M          2           
c1@s2 15360M          2           
[root@magna009 ~]#
Comment 2 Jason Dillaman 2016-04-28 19:11:02 UTC 
Upstream PR: https://github.com/ceph/ceph/pull/8817
Comment 3 Ken Dreyer (Red Hat) 2016-05-10 13:22:12 UTC 
This is undergoing review upstream (https://github.com/ceph/ceph/pull/8869) and will be in v10.2.1.
Comment 4 Ken Dreyer (Red Hat) 2016-05-16 15:30:57 UTC 
The above PR was merged to jewel and is present in v10.2.1.
Comment 7 Hemanth Kumar 2016-05-31 12:09:21 UTC 
I am unable to disable the object-map while the flattening is in-progress

# rbd flatten pool4/bucket11 --cluster master
Image flatten: 69% complete...done.

# rbd feature disable pool4/bucket11 object-map --cluster master
rbd: failed to update image features: (22) Invalid argument2016-05-31 12:01:39.088013 7f5d1fa8fd80 -1 librbd: cannot disable object map

# rbd ls -l bucket10 -p pool4 --cluster master 
NAME             SIZE PARENT FMT PROT LOCK 
bucket10       10240M          2           
bucket10@shot1 10240M          2 yes       
bucket11       10240M          2           
bucket11@shot2 10240M          2           

===========================================================================
Comment 8 Jason Dillaman 2016-05-31 15:26:36 UTC 
@Hamanth: did the image also have fast-diff enabled?
Comment 9 Hemanth Kumar 2016-06-02 10:46:14 UTC 
(In reply to Jason Dillaman from comment #8)
> @Hamanth: did the image also have fast-diff enabled?

Yes Jason. fast-diff was also enabled.
BTW, all the features gets enabled by default with the latest ceph.
Comment 10 Jason Dillaman 2016-06-02 12:46:26 UTC 
@Hemanth: moving this back to ON_QA as it isn't a bug.  Since fast-diff depends on object-map, you cannot disable object-map without also disabling fast-diff.
Comment 11 Hemanth Kumar 2016-06-03 08:09:23 UTC 
(In reply to Jason Dillaman from comment #10)
> @Hemanth: moving this back to ON_QA as it isn't a bug.  Since fast-diff
> depends on object-map, you cannot disable object-map without also disabling
> fast-diff.


@Jason.,
Error messages are still seen while disabling the object map during flattening.


# rbd flatten rbd/image2
Image flatten: 26% complete...2016-06-03 08:01:28.718143 7f3c2ffff700 -1 librbd::object_map::RefreshRequest: failed to load object map: rbd_object_map.17ec5c238e1f29
2016-06-03 08:01:28.718310 7f3c2ffff700 -1 librbd::object_map::InvalidateRequest: 0x7f3c14053500 invalidating object map in-memory
2016-06-03 08:01:28.718322 7f3c2f7fe700 -1 librbd::object_map::InvalidateRequest: 0x7f3c14053500 should_complete: r=0
Image flatten: 100% complete...done.

Disabled the object-map during flattening..
-----------------------------------------
# rbd info image2
rbd image 'image2':
	size 10240 MB in 2560 objects
	order 22 (4096 kB objects)
	block_name_prefix: rbd_data.17ec5c238e1f29
	format: 2
	features: layering, exclusive-lock, object-map, deep-flatten
# rbd feature disable rbd/image2  object-map 
# rbd info image2
rbd image 'image2':
	size 10240 MB in 2560 objects
	order 22 (4096 kB objects)
	block_name_prefix: rbd_data.17ec5c238e1f29
	format: 2
	features: layering, exclusive-lock, deep-flatten
	flags: 

Ceph version :- 10.2.1-7.el7cp.x86_64
Comment 12 Jason Dillaman 2016-06-03 11:26:44 UTC 
Same issue as disabling journal during in-progress operation.  Upstream PR: https://github.com/ceph/ceph/pull/9440
Comment 15 Jason Dillaman 2016-06-10 17:37:09 UTC 
Merged Jewel PR: https://github.com/ceph/ceph/pull/9610
Comment 16 Ken Dreyer (Red Hat) 2016-06-14 16:24:13 UTC 
This is in v10.2.2 upstream, so we will take this change in as part of the rebase to 10.2.2.
Comment 20 errata-xmlrpc 2016-08-23 19:36:29 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1755.html