Description of problem: 1.2.3 -> 1.3.0 upgrade can cause objects with empty names to show up as missing on the 1.3.0 osds Version-Release number of selected component (if applicable): How reproducible: Very. Steps to Reproduce: 1. Create 3 osd 1.2.3 cluster 2. Create an object with an empty name 3. Upgrade one osd to hammer 4. scrub Actual results: Object creation succeeded and scrub turns up inconsistent pg. ceph-osd.0.log:319:2015-08-04 13:30:07.704840 7ff02b984700 -1 log_channel(cluster) log [ERR] : be_compare_scrubmaps: 10.d shard 0 missing bd49d10d//head//10 Note that the object name (bd49d10d//head//10) has an empty object name field. That part is the hallmark of this bug. If the name part is filled in, it's not this bug. Expected results: The initial write of an object with an empty name should have failed, and the subsequent scrub should not have turned up an inconsistent object. Additional info: We'll need to backport to 1.2.3 a fix disallowing this kind of object. This case probably happened due to a radosgw bug which incorrectly creates such an object -- http://tracker.ceph.com/issues/8587.
Original empty named object was created due to ceph issue #8587. The object itself should not exist and can be removed.
If the object shows up due to 8587 (empty name object in .users.swift pool), it can be safely removed since radosgw cannot actually access such an object. If an empty name object shows up in some other pool, one would need to confirm that it's actually garbage before removing it.
You can use rados -p <pool> rm '' to remove the object.
By 'confirm that it's actually garbage' I mean check with someone who understands the application which wrote the object (or the customer if the object is present in a pool used for a custom librados application).
Sage closed the upstream Redmine ticket WONTFIX. Can we do the same here?