You're not really allowed to have /boot as btrfs at present, but custom partitioning UI doesn't filter it out as a possible filesystem type. You can set /boot to btrfs and then complete custom partitioning, but the spoke will show 'error checking storage configuration' and won't let you proceed. Nothing tells you what the problem is, you have to guess. Once you change the fs type of /boot, you can proceed.
According to dlehman, the problem is that the bootloader code doesn't allow a btrfs /boot, but the custom part UI code doesn't have it on the list of disallowed types. We just need to add it to that list. dlehman has a patch ready, this is just bureaucracy.
Proposing as NTH - you can't proceed with installation with the invalid config so it doesn't blow anything up, but it's rather frustrating as you can't actually tell what's wrong, nothing tells you precisely what the problem is.
Discussed at 2012-12-19 NTH review meeting: http://meetbot.fedoraproject.org/fedora-bugzappers/2012-12-19/f18final-blocker-review-6.2012-12-19-17.02.log.txt . Accepted as NTH - this is a simple and safe change that reduces the likelihood of errors and frustration.
Actually, what I was going to do is add btrfs subvol to the list of acceptable grub2 /boot device types. I've just done a test and verified that it works. Others have as well in bug 868465 as well. Are you stuck on preventing it as opposed to fully allowing it?
Nope, not at all, allowing it would be fine. I was just going by what you said on IRC.
this looks like it got stuck in MODIFIED. I checked it the other day and it works.