This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1250205 - 1.2.3 -> 1.3.0 upgrade can cause objects with empty names to show up as missing on the 1.3.0 osds
1.2.3 -> 1.3.0 upgrade can cause objects with empty names to show up as missi...
Status: CLOSED NOTABUG
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RADOS (Show other bugs)
1.3.0
Unspecified Unspecified
unspecified Severity low
: rc
: 1.3.4
Assigned To: Samuel Just
ceph-qe-bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-04 14:02 EDT by Samuel Just
Modified: 2017-07-30 11:13 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-08-03 14:58:34 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Ceph Project Bug Tracker 12610 None None None Never

  None (edit)
Description Samuel Just 2015-08-04 14:02:54 EDT
Description of problem:

1.2.3 -> 1.3.0 upgrade can cause objects with empty names to show up as missing on the 1.3.0 osds


Version-Release number of selected component (if applicable):


How reproducible:

Very.

Steps to Reproduce:
1. Create 3 osd 1.2.3 cluster
2. Create an object with an empty name
3. Upgrade one osd to hammer
4. scrub

Actual results:

Object creation succeeded and scrub turns up inconsistent pg.


ceph-osd.0.log:319:2015-08-04 13:30:07.704840 7ff02b984700 -1 log_channel(cluster) log [ERR] : be_compare_scrubmaps: 10.d shard 0 missing bd49d10d//head//10

Note that the object name (bd49d10d//head//10) has an empty object name field.  That part is the hallmark of this bug.  If the name part is filled in, it's not this bug.

Expected results:

The initial write of an object with an empty name should have failed, and the subsequent scrub should not have turned up an inconsistent object.

Additional info:

We'll need to backport to 1.2.3 a fix disallowing this kind of object.  This case probably happened due to a radosgw bug which incorrectly creates such an object -- http://tracker.ceph.com/issues/8587.
Comment 2 Yehuda Sadeh 2015-08-04 14:14:51 EDT
Original empty named object was created due to ceph issue #8587. The object itself should not exist and can be removed.
Comment 3 Samuel Just 2015-08-04 14:21:52 EDT
If the object shows up due to 8587 (empty name object in .users.swift pool), it can be safely removed since radosgw cannot actually access such an object.  If an empty name object shows up in some other pool, one would need to confirm that it's actually garbage before removing it.
Comment 4 Samuel Just 2015-08-04 14:25:51 EDT
You can use rados -p <pool> rm '' to remove the object.
Comment 5 Samuel Just 2015-08-04 14:57:08 EDT
By 'confirm that it's actually garbage' I mean check with someone who understands the application which wrote the object (or the customer if the object is present in a pool used for a custom librados application).
Comment 6 Ken Dreyer (Red Hat) 2016-08-02 16:56:11 EDT
Sage closed the upstream Redmine ticket WONTFIX. Can we do the same here?

Note You need to log in before you can comment on or make changes to this bug.