Bug 2053276 - [Workload-DFG] ceph orch upgrade get stuck - mgr set_store mon returned -27: error: entry size limited to 65536 bytes
Summary: [Workload-DFG] ceph orch upgrade get stuck - mgr set_store mon returned -27: ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: Cephadm
Version: 5.1
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 6.0
Assignee: Adam King
QA Contact: Manisha Saini
Eliska
URL:
Whiteboard:
Depends On:
Blocks: 2126050
TreeView+ depends on / blocked
 
Reported: 2022-02-10 20:23 UTC by Vikhyat Umrao
Modified: 2023-08-21 14:40 UTC (History)
6 users (show)

Fixed In Version: ceph-17.2.3-12.el9cp
Doc Type: Bug Fix
Doc Text:
.`Cephadm` now splits the device information into multiple entries after exceeding the Ceph monitor store size limit Previously, `cephadm` was unable to refresh the hosts and complete most operations when the device information exceeded the monitor store default maximum size limit of 64K. This caused an entry size error. As a result, users had to raise the default limit if they had hosts with a large number of disks. With this fix, `cephadm` now splits the device information into multiple entries if it takes more space than the size limit. Users no longer have to raise the monitor store entry size limit if they have hosts with a large number of disks.
Clone Of:
Environment:
Last Closed: 2023-03-20 18:55:34 UTC
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Ceph Project Bug Tracker 54251 0 None None None 2022-02-10 20:27:15 UTC
Red Hat Issue Tracker RHCEPH-3144 0 None None None 2022-02-10 20:24:26 UTC
Red Hat Knowledge Base (Solution) 7011273 0 None None None 2023-08-21 14:40:55 UTC
Red Hat Product Errata RHBA-2023:1360 0 None None None 2023-03-20 18:56:02 UTC

Description Vikhyat Umrao 2022-02-10 20:23:41 UTC 
Description of problem:
[Workload-DFG] ceph orch upgrade get stuck - mgr set_store mon returned -27: error: entry size limited to 65536 bytes


Version-Release number of selected component (if applicable):
RHCS 5.0 -> RHCS 5.1
16.2.0-152.el8cp -> 16.2.7-50.el8cp

How reproducible:
Was consistent @Scale with 288 OSD's cluster

Workaround:
ceph config set mon mon_config_key_max_entry_size 1048576

Restart the upgrade process
ceph orch upgrade stop
ceph orch upgrade start --image <image url>
Comment 1 Vikhyat Umrao 2022-02-10 20:24:51 UTC 
The error log message in the active mgr log:

2022-02-10T19:29:14.294+0000 7f91a3331700  0 mgr set_store mon returned -27: error: entry size limited to 65536 bytes. Use 'mon config key max entry size' to manually adjust

# ceph config get mon mon_config_key_max_entry_size
65536

# ceph config set mon mon_config_key_max_entry_size 1048576
# ceph config get mon mon_config_key_max_entry_size
1048576
Comment 26 errata-xmlrpc 2023-03-20 18:55:34 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 6.0 Bug Fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:1360
Note You need to log in before you can comment on or make changes to this bug.