Bug 1483224

Summary: [RHCS 2.y] [RFE] Addition of online osd 'omap'compaction command
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Vikhyat Umrao <vumrao>
Component: RADOSAssignee: Josh Durgin <jdurgin>
Status: CLOSED ERRATA QA Contact: ceph-qe-bugs <ceph-qe-bugs>
Severity: high Docs Contact: Bara Ancincova <bancinco>
Priority: high    
Version: 2.3CC: anharris, ceph-eng-bugs, dzafman, icolle, jdurgin, kchai, kdreyer, tserlin, vakulkar, wusui
Target Milestone: rcKeywords: FutureFeature
Target Release: 2.4   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: RHEL: ceph-10.2.7-45.el7cp Ubuntu: ceph_10.2.7-45redhat1 Doc Type: Enhancement
Doc Text:
.A new `compact` command With this update, the OSD administration socket supports the `compact` command. A large number of `omap` create and delete operations can cause the normal compaction of the `levelDB` database during those operations to be too slow to keep up with the workload. As a result, `levelDB` can grow very large and inhibit performance. The `compact` command compacts the `omap` database (`levelDB` or `RocksDB`) to a smaller size to provide more consistent performance.
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-10-17 18:12:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1473436, 1479701    

Description Vikhyat Umrao 2017-08-19 11:52:22 UTC
Description of problem:

[RHCS 2.y] [RFE] Addition of  online osd 'omap'compaction command

Master branch:
http://tracker.ceph.com/issues/19592
https://github.com/ceph/ceph/pull/16045

Jewel branch:
http://tracker.ceph.com/issues/20675

Version-Release number of selected component (if applicable):
Red Hat Ceph Storage 2.3

Comment 2 Vikhyat Umrao 2017-08-19 19:44:31 UTC
Jewel backport tracker and PR:

http://tracker.ceph.com/issues/20675
https://github.com/ceph/ceph/pull/17101

Comment 14 Warren 2017-10-13 01:26:15 UTC
I am marking this verified because it works on a newly installed system.

However on an osd after I performed the operations in step 10 of https://bugzilla.redhat.com/show_bug.cgi?id=1462011, I got:

[root@magna007 ceph-5]# ceph daemon osd.5 compact
admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
admin_socket: exception getting command descriptions: [Errno 2] No such file or directory

I am not sure if this is a problem or not.  I am also updating bz 1462011 as well.

Comment 15 Warren 2017-10-13 01:45:50 UTC
Has this fix made it into Ubuntu yet?  On my test unbuntu setup, I get:

root@magna024:/var/lib/ceph/osd/ceph-5# ceph daemon osd.5 compact
no valid command found; 10 closest matches:
git_version
getomap <pool> <objectname>
get_command_descriptions
flush_journal
get_latest_osdmap
get_heap_property <property>
dump_ops_in_flight
dump_op_pq_state
dump_watchers
dump_reservations
admin_socket: invalid command
admin_socket: invalid command

Comment 21 Warren 2017-10-16 20:18:43 UTC
Passed on both Ubuntu and Rhel.  I ran into a slight gotcha in that I ran some of the commands as root rather than the ceph user.  Josh helped me a bit and things worked after we changed permissions and ran as the correct user.  This comment also applies to bugzilla #1462011

Comment 23 errata-xmlrpc 2017-10-17 18:12:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2903