Bug 1463969 - bucket check --fix fails with (90) Message too long
Summary: bucket check --fix fails with (90) Message too long
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RGW
Version: 2.3
Hardware: x86_64
OS: Linux
Target Milestone: rc
: 2.4
Assignee: Orit Wasserman
QA Contact: ceph-qe-bugs
Bara Ancincova
Depends On:
Blocks: 1473436 1479701
TreeView+ depends on / blocked
Reported: 2017-06-22 07:45 UTC by Orit Wasserman
Modified: 2017-10-18 18:13 UTC (History)
13 users (show)

Fixed In Version: RHEL: ceph-10.2.7-33.el7cp Ubuntu: ceph_10.2.7-34redhat1
Doc Type: Bug Fix
Doc Text:
.Repairing bucket indexes works as expected Previously, the `cls` method of the Ceph Object Gateway that is used for repairing bucket indexes failed when its output result was too large. Consequently, affected bucket index objects could not be repaired using the `bucket check --fix` command, and the command failed with the "(90) Message too long" error. This update introduces a paging mechanism that ensures that bucket indexes can be repaired as expected.
Clone Of:
Last Closed: 2017-10-17 18:12:51 UTC
Target Upstream Version:

Attachments (Terms of Use)
bucket.py -- create a bucket with some objects (1.08 KB, text/plain)
2017-09-25 16:33 UTC, Warren
no flags Details

System ID Priority Status Summary Last Updated
Ceph Project Bug Tracker 20772 None None None 2017-07-26 07:55:12 UTC
Red Hat Product Errata RHBA-2017:2903 normal SHIPPED_LIVE Red Hat Ceph Storage 2.4 enhancement and bug fix update 2017-10-17 22:12:30 UTC

Description Orit Wasserman 2017-06-22 07:45:57 UTC
Description of problem:
After failure when running bucket rm --purge-objects --bypass-gc  --inconsistent-index. Trying to fix the bucket index resulted in (90) Message too long error from the OSD.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Create a large bucket 300K objects
2. run bucket rm --purge-objects --bypass-gc  --inconsistent-index and stop it in the middle creating a corrupted bucket index
3. try fixing the bucket index by running bucket check --fix

Actual results:
Error 90) Message too long

Expected results:

Additional info:

Comment 20 Warren 2017-09-25 16:33:20 UTC
Created attachment 1330617 [details]
bucket.py -- create a bucket with some objects

Test script used to create files to be deleted.

Comment 21 Warren 2017-09-25 16:38:15 UTC
Using the attached script, 300K files were created in a bucket named foo.

'time radosgw-admin bucket rm --purge-objects --bypass-gc --inconsistent-index --bucket=foo' was run.  This took about 90 minutes.

This process was repeated, and the rm operation was halted after 60 minutes.

radosgw-admin bucket check --fix ran to completion

repeated runs of halted rm operations showed that the number of objects decreased.  Eventually, all files and the bucket were deleted.

At no time did the "Error 90) Message too long" error appear.

Comment 25 Warren 2017-10-13 01:34:15 UTC
This has been verified on Ubuntu as well.

Comment 30 errata-xmlrpc 2017-10-17 18:12:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.