Bug 912154

Summary: btrfs could not do orphan cleanup -28 leaves system unbootable
Product: [Fedora] Fedora Reporter: Kevin Fenzi <kevin>
Component: kernelAssignee: Zach Brown <zab>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 19CC: gansalmon, itamar, jforbes, jonathan, josef, kernel-maint, madhu.chinakonda, sweil
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-04-05 21:13:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
dmesg from netinstall rescue session none

Description Kevin Fenzi 2013-02-17 22:57:14 UTC 
Created attachment 698647 [details]
dmesg from netinstall rescue session

Install was: 

a btrfs volume with "home" and "root" subvolumes encrypted. 
a script that runs hourly and makes a snapshot of each, keeping the last 6 around. 
(installed with the f18 installer)

timeline of events: 

around 2am this morning my time, the system stopped logging. 
around 8am I found the laptop completely unresponsive (blank screen, no drive activity, not response to keys). 
hard power cycled it. 
booted up, but then hung due to being unable to mount / rw. 
I looked at it with dracut emergency and a f18 netsintall iso. 

Trying to mount or use the root subvolume resulted in: 

btrfs: could not do orphan cleanup -28

and the volume was mounted read only. 
remounting rw seemed to work, but any attempt to remove or modify a file resulted in an 'out of space' error. (btrfs df showed 50G+ free)

umounting the root fs resulted in: 

BTRFS: inode 1952590 still on the orphan list

This inode was associated with /var/log/boot.log

also a warning. (see attached dmesg). 

I ran a btrfs-debug-tree on it, and let it run for 30min... I wasn't sure if it was going to complete or not. You can find that output at: 

http://www.scrye.com/~kevin/fedora/debug-tree.xz

The kernel in use was: 3.8.0-0.rc7.git3.1.fc19

I tried 'btrfsck' and 'btrfsck --repair' with no change. It did output issues with some snapshots showing orphans. I tried mounting with 'recover' and 'enospc_debug' without any change. 

I'm happy to try and provide more info, but since this is my primary laptop, I couldn't leave it unusable, so I re-installed with ext4 for now.
Comment 1 Josef Bacik 2013-02-19 15:15:10 UTC 
Sorry about that, we've been fixing these issues as they pop up and I think we've got them all nailed down, they should be going in to 3.9.
Comment 2 Fedora End Of Life 2013-04-03 18:06:09 UTC 
This bug appears to have been reported against 'rawhide' during the Fedora 19 development cycle.
Changing version to '19'.

(As we did not run this process for some time, it could affect also pre-Fedora 19 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 19 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora19
Comment 3 Justin M. Forbes 2013-04-05 17:00:48 UTC 
Is this still a problem with 3.9 based F19 kernels?
Comment 4 Kevin Fenzi 2013-04-05 20:44:31 UTC 
Sadly, I don't know. ;( 

I reinstalled with ext4... sorry.