Bug 1479949 - ‘radosgw-admin bucket check –fix’ deletes indices for valid objects as well
‘radosgw-admin bucket check –fix’ deletes indices for valid objects as well
Status: CLOSED ERRATA
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RGW (Show other bugs)
2.3
x86_64 Linux
high Severity high
: rc
: 2.4
Assigned To: Matt Benjamin (redhat)
ceph-qe-bugs
Bara Ancincova
:
Depends On:
Blocks: 1473436 1479701
  Show dependency treegraph
 
Reported: 2017-08-09 15:39 EDT by Benjamin Schmaus
Modified: 2017-10-18 14:14 EDT (History)
15 users (show)

See Also:
Fixed In Version: RHEL: ceph-10.2.7-33.el7cp Ubuntu: ceph_10.2.7-34redhat1
Doc Type: Bug Fix
Doc Text:
.Fixing bucket indexes no longer damages them Previously, a bug in the Ceph Object Gateway namespacing could cause the bucket index repair process to incorrectly delete object entries. As a consequence, an attempt to fix a bucket index could damage the index. The bug has been fixed, and fixing bucket indexes no longer damages them.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-10-17 14:12:51 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
bucket.py -- create a bucket with some objects (1.08 KB, text/plain)
2017-09-22 20:43 EDT, Warren
no flags Details
testmulti.py -- this creates an unfinished multpart uploaded file (1.93 KB, text/plain)
2017-09-22 20:45 EDT, Warren
no flags Details
testmulti.py -- multisite tester. (1.93 KB, text/plain)
2017-09-22 20:46 EDT, Warren
no flags Details
viewbucket.py -- look at bucket objects. (525 bytes, text/plain)
2017-09-22 20:48 EDT, Warren
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Ceph Project Bug Tracker 18470 None None None 2017-08-09 15:39 EDT

  None (edit)
Description Benjamin Schmaus 2017-08-09 15:39:30 EDT
Description of problem:

Running the following command: radosgw-admin bucket check –fix

In builds of Jewel >= 10.2.7 results in valid objects being removed as well.

Upstream master has been fixed but we need a fix for Ceph 2.4.

http://tracker.ceph.com/issues/18470

https://github.com/ceph/ceph/pull/16856
	
Version-Release number of selected component (if applicable):
2.3

How reproducible:
100%

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 2 Ian Colle 2017-08-16 17:42:38 EDT
Moving to 2.5 until post 2.4 release.
Comment 3 shilpa 2017-09-11 03:52:35 EDT
@matt, @orit,

Could you please suggest a reproducer for this?
Comment 6 Orit Wasserman 2017-09-11 05:39:32 EDT
(In reply to shilpa from comment #3)
> @matt, @orit,
> 
> Could you please suggest a reproducer for this?

This is bug caused bucket check to remove non multipart entries.
You can try creating a bucket, adding objects.
Upload a multipart objects at least one that is not commited.
Run bucket check --fix see all the objects are there
Comment 7 shilpa 2017-09-11 05:48:26 EDT
(In reply to Orit Wasserman from comment #6)
> (In reply to shilpa from comment #3)
> > @matt, @orit,
> > 
> > Could you please suggest a reproducer for this?
> 
> This is bug caused bucket check to remove non multipart entries.
> You can try creating a bucket, adding objects.
> Upload a multipart objects at least one that is not commited.
> Run bucket check --fix see all the objects are there

Thanks Orit. Do you mean initiate a multipart upload that is incomplete without doing a complete_upload?
Comment 8 Orit Wasserman 2017-09-11 06:38:43 EDT
(In reply to shilpa from comment #7)
> (In reply to Orit Wasserman from comment #6)
> > (In reply to shilpa from comment #3)
> > > @matt, @orit,
> > > 
> > > Could you please suggest a reproducer for this?
> > 
> > This is bug caused bucket check to remove non multipart entries.
> > You can try creating a bucket, adding objects.
> > Upload a multipart objects at least one that is not commited.
> > Run bucket check --fix see all the objects are there
> 
> Thanks Orit. Do you mean initiate a multipart upload that is incomplete
> without doing a complete_upload?

yes
Comment 12 Warren 2017-09-22 20:43 EDT
Created attachment 1329785 [details]
bucket.py -- create a bucket with some objects

script used to create a bucket with some objects in it.
Comment 13 Warren 2017-09-22 20:45 EDT
Created attachment 1329786 [details]
testmulti.py -- this creates an unfinished multpart uploaded file

Creates a multpart uploaded file that is not complete.
Comment 14 Warren 2017-09-22 20:46 EDT
Created attachment 1329787 [details]
testmulti.py -- multisite tester.

This creates an incomplete multipart file.
Comment 15 Warren 2017-09-22 20:48 EDT
Created attachment 1329788 [details]
viewbucket.py -- look at bucket objects.

This is used to verify that other objects in the bucket are not affected by the check run on the multipart object.
Comment 16 Warren 2017-09-22 20:55:06 EDT
Test procedure:

1. cat a /var/log/*/* file that was big with itself repeatedly until you get a 16G text file named bigstuff.

2. run python bucket.py foo 1 to create a bucket with 1000 objects in it.

3. run python testmulti.py foo bigstuff

4. run radosgw-admin bucket check --fix --bucket=foo

5. run python viewbucket.py foo to make sure that bigstuff is gone and all the other files are still there.

This test reproduced the bug on 2.4 and the bug did not occur on 2.4a
Comment 17 Warren 2017-10-10 20:11:37 EDT
(In reply to Warren from comment #14)
> Created attachment 1329787 [details]
> testmulti.py -- multisite tester.
> 
> This creates an incomplete multipart file.

Ignore this comment. It inaccurately repeats the one before.
Comment 18 Warren 2017-10-10 21:51:46 EDT
Verified on Ubuntu
Comment 23 Warren 2017-10-12 21:32:24 EDT
This has been verified on Ubuntu as well.
Comment 25 errata-xmlrpc 2017-10-17 14:12:51 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2903

Note You need to log in before you can comment on or make changes to this bug.