Bug 1463969

Summary: bucket check --fix fails with (90) Message too long
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Orit Wasserman <owasserm>
Component: RGWAssignee: Orit Wasserman <owasserm>
Status: CLOSED ERRATA QA Contact: ceph-qe-bugs <ceph-qe-bugs>
Severity: medium Docs Contact: Bara Ancincova <bancinco>
Priority: high    
Version: 2.3CC: anharris, cbodley, ceph-eng-bugs, icolle, kbader, mbenjamin, owasserm, sweil, tserlin, uboppana, vakulkar, vumrao, wusui
Target Milestone: rc   
Target Release: 2.4   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: RHEL: ceph-10.2.7-33.el7cp Ubuntu: ceph_10.2.7-34redhat1 Doc Type: Bug Fix
Doc Text:
.Repairing bucket indexes works as expected Previously, the `cls` method of the Ceph Object Gateway that is used for repairing bucket indexes failed when its output result was too large. Consequently, affected bucket index objects could not be repaired using the `bucket check --fix` command, and the command failed with the "(90) Message too long" error. This update introduces a paging mechanism that ensures that bucket indexes can be repaired as expected.
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-10-17 18:12:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1473436, 1479701    
Attachments:
Description Flags
bucket.py -- create a bucket with some objects none

Description Orit Wasserman 2017-06-22 07:45:57 UTC
Description of problem:
After failure when running bucket rm --purge-objects --bypass-gc  --inconsistent-index. Trying to fix the bucket index resulted in (90) Message too long error from the OSD.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Create a large bucket 300K objects
2. run bucket rm --purge-objects --bypass-gc  --inconsistent-index and stop it in the middle creating a corrupted bucket index
3. try fixing the bucket index by running bucket check --fix

Actual results:
Error 90) Message too long

Expected results:
success

Additional info:

Comment 20 Warren 2017-09-25 16:33:20 UTC
Created attachment 1330617 [details]
bucket.py -- create a bucket with some objects

Test script used to create files to be deleted.

Comment 21 Warren 2017-09-25 16:38:15 UTC
Using the attached script, 300K files were created in a bucket named foo.

'time radosgw-admin bucket rm --purge-objects --bypass-gc --inconsistent-index --bucket=foo' was run.  This took about 90 minutes.

This process was repeated, and the rm operation was halted after 60 minutes.

radosgw-admin bucket check --fix ran to completion

repeated runs of halted rm operations showed that the number of objects decreased.  Eventually, all files and the bucket were deleted.

At no time did the "Error 90) Message too long" error appear.

Comment 25 Warren 2017-10-13 01:34:15 UTC
This has been verified on Ubuntu as well.

Comment 30 errata-xmlrpc 2017-10-17 18:12:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2903