Bug 1250205

Summary:	1.2.3 -> 1.3.0 upgrade can cause objects with empty names to show up as missing on the 1.3.0 osds
Product:	[Red Hat Storage] Red Hat Ceph Storage	Reporter:	Samuel Just <sjust>
Component:	RADOS	Assignee:	Samuel Just <sjust>
Status:	CLOSED NOTABUG	QA Contact:	ceph-qe-bugs <ceph-qe-bugs>
Severity:	low	Docs Contact:
Priority:	unspecified
Version:	1.3.0	CC:	ceph-eng-bugs, dzafman, flucifre, kchai, kdreyer, sjust, vumrao, yehuda
Target Milestone:	rc
Target Release:	1.3.4
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-08-03 18:58:34 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Samuel Just 2015-08-04 18:02:54 UTC

Description of problem:

1.2.3 -> 1.3.0 upgrade can cause objects with empty names to show up as missing on the 1.3.0 osds


Version-Release number of selected component (if applicable):


How reproducible:

Very.

Steps to Reproduce:
1. Create 3 osd 1.2.3 cluster
2. Create an object with an empty name
3. Upgrade one osd to hammer
4. scrub

Actual results:

Object creation succeeded and scrub turns up inconsistent pg.


ceph-osd.0.log:319:2015-08-04 13:30:07.704840 7ff02b984700 -1 log_channel(cluster) log [ERR] : be_compare_scrubmaps: 10.d shard 0 missing bd49d10d//head//10

Note that the object name (bd49d10d//head//10) has an empty object name field.  That part is the hallmark of this bug.  If the name part is filled in, it's not this bug.

Expected results:

The initial write of an object with an empty name should have failed, and the subsequent scrub should not have turned up an inconsistent object.

Additional info:

We'll need to backport to 1.2.3 a fix disallowing this kind of object.  This case probably happened due to a radosgw bug which incorrectly creates such an object -- http://tracker.ceph.com/issues/8587.

Comment 2 Yehuda Sadeh 2015-08-04 18:14:51 UTC

Original empty named object was created due to ceph issue #8587. The object itself should not exist and can be removed.

Comment 3 Samuel Just 2015-08-04 18:21:52 UTC

If the object shows up due to 8587 (empty name object in .users.swift pool), it can be safely removed since radosgw cannot actually access such an object.  If an empty name object shows up in some other pool, one would need to confirm that it's actually garbage before removing it.

Comment 4 Samuel Just 2015-08-04 18:25:51 UTC

You can use rados -p <pool> rm '' to remove the object.

Comment 5 Samuel Just 2015-08-04 18:57:08 UTC

By 'confirm that it's actually garbage' I mean check with someone who understands the application which wrote the object (or the customer if the object is present in a pool used for a custom librados application).

Comment 6 Ken Dreyer (Red Hat) 2016-08-02 20:56:11 UTC

Sage closed the upstream Redmine ticket WONTFIX. Can we do the same here?