Bug 870207

Oh, I missed one other point Ric made: we still have a plan to switch to btrfs-by-default for the long term in the near future, probably at F19 or at latest F20.

If we switch to raw-ext4-by-default for just one or two releases, we make a significant change to our behaviour for a short time and then change it *again*. This has a couple of drawbacks:

1) We have a body of existing documentation for LVM operations in the installation guide and other places, we don't have this for raw ext4. It would be quite some effort to update it all for ext4 and then update it all _again_ for btrfs.

2) When it comes to supporting existing installs and upgrades, we already have enough different configurations to contend with. LVM-by-default for FC3->F17, ext4-by-default for F18 and possibly F19, and btrfs-by-default for F19 or F20 onwards gives us three 'common' default configurations we always have to keep in mind for support purposes. If we just stick with LVM-by-default until we switch to btrfs, we only have two 'common' default configurations to worry about.

(In reply to comment #1)
> If we just stick with LVM-by-default until we switch to btrfs,
> we only have two 'common' default configurations to worry about.

Please do not ignore disk encryption. If there is full support in btrfs, fine. But I am not sure they are working on it, so you can end up with btrfs over dmcrypt...

NACK with a HAMMER this is not something that was not known before.

For anyone interested in checking out the potential change, http://adamwill.fedorapeople.org/updates-lvm.img implements it, using the patch dlehman provided.

(In reply to comment #3)
> NACK with a HAMMER this is not something that was not known before.

Actually, following the thread, to the large extent it was not known, at least some FESCO members claim they were not aware of it when NewUI was approved.

First of all this it's to dam late in the release cycle to change this now and if we change this it means we have to slip another week to properly test anaconda with lvm as default against the alpha and beta criteria. 

And as has been mentioned on numerous occasion and drago01 reiterates on the test list

"What exactly does not hold anymore? Resizing partitions isn't that
common and not the primary use of LVM (you can do it without it and
most users won't).  It is still pretty much useless (as in the extra
features won't be used) for the average desktop / laptop installs. For
most users all it does is slowing down the boot process (we should
stop adding crap to the default boot process because someone might
need it on some obscure case). Those who know about the extra features
and want to use them will enable it anyway regardless on what we set
as defaults."

What triggered this request in the first place RHEL needs or the recent EXT4 bug?

Certainly not the ext4 bug. It's coming from RH devs but it's not really 'RHEL needs' - RHEL could change the default for itself if it wanted to, and that was the expectation all along (the code has been designed such that different distros that use anaconda can have different autopart defaults).

But Ric and others, when they became aware of the change, just didn't think it was the right thing _for Fedora_ either. They didn't think there was any decent justification for changing the default away from LVM and were concerned that the change wasn't well communicated (it wasn't mentioned in the feature page).

I think we can just argue it on the merits, through this bug and/or the ML threads. And the lateness issue is obviously a big concern too.

On the feature front, snapper [1] is one example of an upcoming tool that requires an installation with either LVM or btrfs.  It would be a pity to deny this to users who had chosen the default installation.

Separately, work is also in progress to use SSDs to cache data from slower disks and this too needs LVM.

[1] http://snapper.io/ (review request bug 852174)

Any storage admin with access to high end storage devices knows that manage storage in GNU/Linux sucks you cant do as simple task as just expanding a disk from the storage management interface and be done with it and afaik even today lvm lacks reorgvg... 

The bottom line is that the novice enduser never will use the "advanced" enterprise features of lvm ( or btrfs for that matter ) and every single admin I know does not use the "default" partitioning scheme of anaconda I know they all customizes the partitioning scheme ( admins want to keep /var or more to the point /var/log on separate partition to prevent it to accidental fill the filesystem for example ). 

So whom is Anaconda trying to target here with the default? 

The novice enduser that expects the installer to provide him with the best optimal enduser experience for his desktop install or the those RH devs because the default partitioning scheme in no way suits administrators need and never can due to the various need and complexity in infrastructures..

(In reply to comment #9)
> On the feature front, snapper [1] is one example of an upcoming tool that
> requires an installation with either LVM or btrfs.  It would be a pity to
> deny this to users who had chosen the default installation.
> 
> Separately, work is also in progress to use SSDs to cache data from slower
> disks and this too needs LVM.
> 
> [1] http://snapper.io/ (review request bug 852174)

Are you really claiming here that the novice enduser will ever use "snapper" and those that will cant choose to install it at install time if they want or need lvm or btrfs?

Adam asked me to add this to the bug so:

 don't have current numbers but it adds a synchronisation point (i.e
breaks a fully parallel bootup) let me just quote this
(http://0pointer.de/blog/projects/blame-game.html):

"Let's have a closer look at the worst offender on this boot: a
service by the name of udev-settle.service. So why does it take that
much time to initialize, and what can we do about it? This service
actually does very little: it just waits for the device probing being
done by udev to finish and then exits. Device probing can be slow.
[..]  So, why is udev-settle.service part of our boot process? Well,
it actually doesn't need to be. It is pulled in by the storage setup
logic of Fedora: to be precise, by the *LVM*, RAID and Multipath setup
script. These storage services have not been implemented in the way
hardware detection and probing work today: they expect to be
initialized at a point in time where "all devices have been probed",
so that they can simply iterate through the list of available disks
and do their work on it. However, on modern machinery this is not how
things actually work: hardware can come and hardware can go all the
time, during boot and during runtime. For some technologies it is not
even possible to know when the device enumeration is complete
(example: USB, or iSCSI), thus waiting for all storage devices to show
up and be probed must necessarily include a fixed delay when it is
assumed that all devices that can show up have shown up, and got
probed. In this case all this *shows very negatively in the boot
time*: the storage scripts force us to delay bootup until all
potential devices have shown up and all devices that did got probed"

I should also mention from nooob usability issue you can take an ext4 formatted disk and plug him in any other linux distribution and start work on it plug and pray end user experience. You cant do so with LVM without jumping through massive usability hoops and command line foo..

Hmm, does LVM still pull in udev-settle and scsi-wait-scan? If so, it's a major source of slowness during boot, and hence a bad candidate to stick into the default boot of all systems.

There exist 3rd party drivers to access ext2/3/4 on other operating system like Microsoft Windows and OS-X afaik none exist to access lvm...

(In reply to comment #13)
> I should also mention from nooob usability issue you can take an ext4
> formatted disk and plug him in any other linux distribution and start work
> on it plug and pray end user experience. You cant do so with LVM without
> jumping through massive usability hoops and command line foo..

Strange, I am doing it all the time for several years and it works.
Yes, it is rocket science, namely to type "vgchange -a y" (and it can be of course automated like LUKS.)

(In reply to comment #11)
> Are you really claiming here that the novice enduser will ever use "snapper"
> and those that will cant choose to install it at install time if they want
> or need lvm or btrfs?

I think some people who select the defaults will later want to use features like these and be disappointed to find this means reinstalling (or copying their filesystem elsewhere, reconfiguring and copying it back).

"You have plenty of space to install Fedora, so we can automatically configure the rest of the installation for you."

That sounds to me to be encouraging selection by a far wider group than just "novice endusers".  Today's novice is tomorrow's advanced user.

Isn't the wiser option the way it used to be, with these capabilities enabled by default for the novice?  Those who are confident they won't ever need them can still turn them off.

Lennart: "Hmm, does LVM still pull in udev-settle and scsi-wait-scan? If so, it's a major source of slowness during boot, and hence a bad candidate to stick into the default boot of all systems." - well, from what I can see in the LVM test install I just did, fedora-storage-init.service (description "Initialize storage subsystems (RAID, LVM, etc.)") Wants fedora-wait-storage.service , which Wants systemd-udev-settle.service and itself modprobes scsi_wait_scan. So it seems like that's the case, from my knowledge anyway.

(In reply to comment #16)
> (In reply to comment #13)
> > I should also mention from nooob usability issue you can take an ext4
> > formatted disk and plug him in any other linux distribution and start work
> > on it plug and pray end user experience. You cant do so with LVM without
> > jumping through massive usability hoops and command line foo..
> 
> Strange, I am doing it all the time for several years and it works.
> Yes, it is rocket science, namely to type "vgchange -a y" (and it can be of
> course automated like LUKS.)

Yes and you failed to understand the noob does not know how to find his way in the terminal let alone be an expert to run the above mentioned command and let alone that he's an expert if he needs "rename" that volume group or other issue that might arise when he tries to access that disk. He does not have to do any of that with ext2/3/4 he just plugs the disk in and voila...

(In reply to comment #17)
> (In reply to comment #11)
> > Are you really claiming here that the novice enduser will ever use "snapper"
> > and those that will cant choose to install it at install time if they want
> > or need lvm or btrfs?
> 
> I think some people who select the defaults will later want to use features
> like these and be disappointed to find this means reinstalling (or copying
> their filesystem elsewhere, reconfiguring and copying it back).

There is absolutely no guarantee or even remote one that that will happen which renders this argument mood.

> 
> "You have plenty of space to install Fedora, so we can automatically
> configure the rest of the installation for you."
> 
> That sounds to me to be encouraging selection by a far wider group than just
> "novice endusers".  Today's novice is tomorrow's advanced user.

This argument does not hold water either and to remotely do so the novice end user must have the best user experience from the start for him to want to a) use your distribution b) explore it further. 

> Isn't the wiser option the way it used to be, with these capabilities
> enabled by default for the novice?  Those who are confident they won't ever
> need them can still turn them off.

No the wisest chose is to default with the best settings aimed at the lowest dominator in this case the novice end user. Advanced and expert users can take care of themselves...

(In reply to comment #20)
> No the wisest chose is to default with the best settings aimed at the lowest
> dominator in this case the novice end user. Advanced and expert users can
> take care of themselves...

To summarise my position on the "novice" point:

The best settings for the novice include having LVM enabled by default in case that novice later discovers they do need the features.

Without the preparation at install time, they are excluded from using anything that depends upon those features in future without performing major surgery on their system (like a reinstall).

Advanced and expert users who already know they don't need or want LVM do not need to include it in their installation.

(In reply to comment #21)
> (In reply to comment #20)
> > No the wisest chose is to default with the best settings aimed at the lowest
> > dominator in this case the novice end user. Advanced and expert users can
> > take care of themselves...
> 
> To summarise my position on the "novice" point:
> 
> The best settings for the novice include having LVM enabled by default in
> case that novice later discovers they do need the features.
> 
> Without the preparation at install time, they are excluded from using
> anything that depends upon those features in future without performing major
> surgery on their system (like a reinstall).
> 
> Advanced and expert users who already know they don't need or want LVM do
> not need to include it in their installation.

So you want to deliver worse initial user experience for the few that *might* want to use lvm later hey then why dont we default now to btrfs since those users might want to take advantage of all the bells and whistle it has to offer encase they decide to upgrade instead of doing a fresh install if and when btrfs finally becomes the default filesystem in Fedora...

And on the upgrade topic afaik it's easier to migrate from ext4 to btrfs then migrating from lvm to ext4 so based on that defaulting to ext4 even if for only one or two release cycle is better then defaulting to lvm...

From all I've heard of it I don't think we're likely to be offering migration of existing FSes of *any* type to btrfs or encouraging anyone to do it. AFAIK it's theoretically possible to migrate ext4 to btrfs but it's really not a good idea and shouldn't be encouraged.

Remember, when we talk about 'LVM' we're talking about 'ext4 on LVM'. The partitions are ultimately still ext4 partitions. Though I don't know if anyone would really want to be running btrfs on LVM, presumably it'd be theoretically possible...

Regarding the udev settle point, we do now have an LVM mode that doesn't require this any more.  I'll check with the team tomorrow, but I think we could probably enable this mode by default now in f18, if that's helpful.

(In reply to comment #24)
> From all I've heard of it I don't think we're likely to be offering
> migration of existing FSes of *any* type to btrfs or encouraging anyone to
> do it. AFAIK it's theoretically possible to migrate ext4 to btrfs but it's
> really not a good idea and shouldn't be encouraged.
> 
> Remember, when we talk about 'LVM' we're talking about 'ext4 on LVM'. The
> partitions are ultimately still ext4 partitions. Though I don't know if
> anyone would really want to be running btrfs on LVM, presumably it'd be
> theoretically possible...

I was talking about tearing down lvm then migrate the ext4 to btrfs not lvm on btrfs that makes absolutely no sense.. 

If FESCO is going to be smart about defaulting to btrfs we should do it in F20 and default to ext4 from now on to allow as smooth transaction to btrfs.

The btrfs-convert utility should be included with the installer in this release cycle so it gives us and others enough time to test conversion from rescue mode and work ahead of us in QA for once and being able to work in advance instead of being stuck in whole another installation/migration/upgrade mess atleast that's my pov... 

Atleast this is the time FESCO should seriously looking into btrfs as an default and implement what is needed to allow for as smooth transaction to it as possible with upgrade support in mind ( you know since we officially support it ) 

I dont think the anaconda team can avoid supporting upgrading to btrfs both in the installer and or the fedup because either the installer or the upgrade tool is the best place to do it...

johann: I get what you're saying, but my understanding is that ext4->btrfs migration is a dangerous and generally-not-recommended operation, and we wouldn't be recommending people migrate existing ext4 partitions to btrfs when we start defaulting to btrfs, nor would we be automatically migrating ext4 partitions to btrfs as part of upgrade: we'd just be leaving ext4 partitions as ext4 for upgraders, only new installs would be getting btrfs. So the idea that by going to ext4 by default we 'prepare the way' for a migration to btrfs doesn't quite fly. I might be wrong about this, if a btrfs expert wants to correct me, please do.

(In reply to comment #25)
> Regarding the udev settle point, we do now have an LVM mode that doesn't
> require this any more.  I'll check with the team tomorrow, but I think we
> could probably enable this mode by default now in f18, if that's helpful.

I've been in an (entirely unrelated) discussion with Peter Rajinoha about this - lvmetad exists in F-18, and he has patches staged for testing in scratch build that completely eliminate the fedora-storage-init script, regardless of whether lvmetad is used or not.

So, my understanding of this bug is that the default was changed from ext4-on-LVM to raw ext4 as a poorly communicated part of a much larger feature (i.e. the general installer UI updates).

While this change was made for reasonable UX reasons, it *was not communicated* in a way that allowed the storage experts to realise it was being done, and deliberately debate the pros and cons of doing so.

Some of the tone that seems to be coming across now that this has been pointed out is "It's too late in the F18 cycle to change it now, so we're going to keep the new default behaviour. We don't care that that this change was effectively slipped by many of the storage experts by stealth, as the fact we only accidentally excluded them from the discussion rather than doing so on purpose makes it all fine". Surely the reasonable thing to do after discovering that a relevant segment of the community was accidentally left out of a key technical discussion is to revert to the F17 default for F18, and then discuss the topic of changing the default install behaviour *explicitly* for F19, subject to direct FESCo approval, rather than as an implicit part of a much larger change?

What harm is there in being conservative with this and sticking with the F17 behaviour for another cycle? Especially if the storage folks can adjust the settings in LVM to eliminate the harmful effects on the boot time?

(Also, if you want a simple use case that I find LVM makes much, much easier: as a desktop user, install a new harddrive and use it to allocate more space to /home)

nick: the harm is that, right now, we are one week from signing off on Fedora 18 Beta. Fedora runs on very tight cycles. In theory this is a small and safe change, but we've had problems with small and safe changes before :), and Fedora timescales don't give us much wiggle room for dealing with problems. Everyone dealing with Fedora releases has learned to be leery of making changes to the code we've been testing and working with so far, at this late point.

For reference for those who don't follow Fedora stuff closely, the Fedora Beta freeze happened this morning, and the Go/No-Go meeting is scheduled for 2012-11-01; we have to have a fully-tested and signed off release candidate spun by that date in order to release Beta without further slips. Making changes to partitioning behaviour at this time obviously potentially affects our ability to do that.

It's a tricky problem because I can entirely see both sides of the argument. You are entirely right about the problems with how the change was handled in the first place, and Johann is entirely right to be worried about the consequences of changing this 'back' this late in the Fedora cycle. It's kind of hard to see which 'right' we should pick.

I don't think it's too late to switch it back to lvm-by-default for F18 if that's what Fedora wants. We just have to flip the switch before the beta goes out.

(In reply to comment #28)
 
> I've been in an (entirely unrelated) discussion with Peter Rajinoha about
> this - lvmetad exists in F-18, and he has patches staged for testing in
> scratch build that completely eliminate the fedora-storage-init script,
> regardless of whether lvmetad is used or not.

That sounds encouraging.

man lvmetad:

       By  default,  lvmetad,  even  if  running,  is  not  used  by  LVM. See
       lvm.conf(5).

That, not so much...

For the record I've done some testing with the updates.img I provided above. I've done a non-custom-part install and a custom part install using the 'create partitions automatically' option in custom part, in a KVM with a previously-populated disk (wiping all the partitions as part of the install), and both worked fine, which is pretty much what I expected given that this is well-known code.

But it's always _theoretically_ possible we've missed something. For instance, the names of the LVs are different in F18 than they were in F17, I think - they're called 'fedora-root' and 'fedora-home', not LogVol00 or whatever the heck it was before. That might affect something somewhere. There's always a risk to any change.

And, to follow up on comment 32, lvm.conf(5) contains no hint that I could find about how to make lvm use lvmetad...

Matthias - there's a use_lvmetad option, IIRC. That being said, it's likely somewhat tangential to this issue.

(In reply to comment #28)
> (In reply to comment #25)
> > Regarding the udev settle point, we do now have an LVM mode that doesn't
> > require this any more.  I'll check with the team tomorrow, but I think we
> > could probably enable this mode by default now in f18, if that's helpful.
> 
> I've been in an (entirely unrelated) discussion with Peter Rajinoha about
> this - lvmetad exists in F-18, and he has patches staged for testing in
> scratch build that completely eliminate the fedora-storage-init script,
> regardless of whether lvmetad is used or not.

Yes, the intention of this change is to clean up the fedora-storage-init script. 

This one currently encompasses multipath, dmraid, mdraid and lvm2 and as a whole it depends on udev-settle these days. However, the mpath and mdraid do not require udev-settle already (as discussed with maintainers of these parts - it's been just a fallback in case the activation is not done correctly the udev-event-based way) and lvm2 does not require it as well with lvmetad used (global/use_lvmetad=1 lvm.conf setting).

Then the only part remaining that still requires the udev-settle is the dmraid. However, as all of these will now be in their own separate package, the udev-settle will be brought in only in case dmraid is installed and used (and dmraid is not a very common package to use anyway as it solves specific situation only I think a common user is not interested in) and only if lvmetad is not used. And lvmetad will become enabled by default. So finally, the majority of storage activation does not depend on udev-settle. This is not an issue anymore.

(In reply to comment #29)

> What harm is there in being conservative with this and sticking with the F17
> behaviour for another cycle? Especially if the storage folks can adjust the
> settings in LVM to eliminate the harmful effects on the boot time?

You'd have to provide numbers to back it up. Not that claim that this is the case because if it turned out not to be we'd be stuck with it for F18.
 
> (Also, if you want a simple use case that I find LVM makes much, much
> easier: as a desktop user, install a new harddrive and use it to allocate
> more space to /home)

Which doubles the probability of a disk failure causing your data to be lost (just like RAID0).

(In reply to comment #33)
> For the record I've done some testing with the updates.img I provided above.
> I've done a non-custom-part install and a custom part install using the
> 'create partitions automatically' option in custom part, in a KVM with a
> previously-populated disk (wiping all the partitions as part of the
> install), and both worked fine, which is pretty much what I expected given
> that this is well-known code.
> 
> But it's always _theoretically_ possible we've missed something. For
> instance, the names of the LVs are different in F18 than they were in F17, I
> think - they're called 'fedora-root' and 'fedora-home', not LogVol00 or
> whatever the heck it was before. That might affect something somewhere.
> There's always a risk to any change.

For example did you test this on vm only or did you do so on a physical hw and if so on what types of physical hw ( non raid, fake raid real raid sw raid iscsi etc ) etc..

We cant help if those RH devs and their storage community ( I'm certanly not aware of any storage community here in Fedora )where somewhere vacationing on bora bora drinking pina colada while this change was made. 

No more that those that have noticed serious design flaws with the newUI weren't there to attend fudcon comment on Duffy's blogs, monitoring the design or the anaconda list when those discussions took place...

Hi Johann,

Not sure why you think that there is no storage community in Fedora. Ext4, btrfs, ceph, pNFS were all brought live in Fedora first and done by our kernel developers.

Maybe we do our job so well that you just did not notice us :)

We do a ton of work in Fedora and it is required as a rule for all of the Red Hat paid FS developers to push code upstream, integrate in Fedora before we can even consider bringing it into a RHEL release.

To the point of the request. Changing the default which we have had in Fedora from installing on LVM to installing on a raw partition has several large impacts on Fedora users:

* it is a stop gap as a default since we plan to have F19 default to btrfs (this will bring in a small set of new machines with ext4 on raw with most people continuing to have ext4 on LVM from previous installs)

* there are desktop features and fedora feature pages that depend on LVM (or btrfs) that are approved Fedora features (storage system manager, the LVM plugin to do rollback after a bad upgrade) and upstream projects like dm-cache that will allow desktop users (with an SSD and a hard disk) to use pure open source stacks to do caching

* there is *NO* Fedora specific documentation on how to do sys admin tasks. We do have RHEL documentation, Fedora documentation and community experience on how to admin boxes on top of LVM.

* this change was done in good faith as part of the UI redesign, but that was not highlighted in the release as a significant change.

In short, as the storage and file system people who have to support Fedora, this is a make work default that will take away desktop functionality from our fedora users and will be supplanted by btrfs (without LVM) in F19.

Not good for users, not good for the Fedora developers who support storage and file systems.

Let's just keep the default in place and stay focused on the btrfs work.

Thanks!

One separate note.

Converting in place from ext4 to btrfs OR in place from ext3 to ext4 is a *really* bad and silly thing to do for a ton of reasons.

It kind of appears to work, no one tests it, and when you report an upstream bug about it, you won't get support (more chuckles).

It was designed as a toy to let you play on a file system that you don't care about, not the preferred way to install a btrfs file system (or move to ext4 from ext3).

thanks!

(In reply to comment #39)
> Hi Johann,
> 
> Not sure why you think that there is no storage community in Fedora. Ext4,
> btrfs, ceph, pNFS were all brought live in Fedora first and done by our
> kernel developers.
> 
> Maybe we do our job so well that you just did not notice us :)

I missed the whole newUI discussion so I would not be surprised that I did not notice you. Where does that community reside exactly? 

> We do a ton of work in Fedora and it is required as a rule for all of the
> Red Hat paid FS developers to push code upstream, integrate in Fedora before
> we can even consider bringing it into a RHEL release.
> 

That's Red Hat and Red Hat != Fedora 

> To the point of the request. Changing the default which we have had in
> Fedora from installing on LVM to installing on a raw partition has several
> large impacts on Fedora users:
> 
> * it is a stop gap as a default since we plan to have F19 default to btrfs
> (this will bring in a small set of new machines with ext4 on raw with most
> people continuing to have ext4 on LVM from previous installs)
> 
> * there are desktop features and fedora feature pages that depend on LVM (or
> btrfs) that are approved Fedora features (storage system manager, the LVM
> plugin to do rollback after a bad upgrade) and upstream projects like
> dm-cache that will allow desktop users (with an SSD and a hard disk) to use
> pure open source stacks to do caching

So you are saying that without defaulting to LVM everything breaks and wont work? 

Or is the case here simply they cant take advantage of option that the majority of Fedora user base wont likely never use anyway...

> * there is *NO* Fedora specific documentation on how to do sys admin tasks.
> We do have RHEL documentation, Fedora documentation and community experience
> on how to admin boxes on top of LVM.

Arguably FS directly on a partition has way more documentation then anything else out here so I'm not following you here. 

> * this change was done in good faith as part of the UI redesign, but that
> was not highlighted in the release as a significant change.

Yea that and a bunch of other stuff they lacked to mention in anycase this is less about what's the default and more about the *when* the change being introduced.

> In short, as the storage and file system people who have to support Fedora,
> this is a make work default that will take away desktop functionality from
> our fedora users and will be supplanted by btrfs (without LVM) in F19.

It's irrelevant to the project what ever corporate policy employees of the said corporate have to follow and where do you get that F19 will be default to BTRFS?

Have you guys just decided that for FESCO and the project in whole?

And last time I spoke with Josef he mentioned ( which admittedly was a while back ) that it probably would not be *properly* ready until F20.

Has that changed?
 
Have the filesystem repair tools appeared etc. or are you guys simply marketing BTRFS like they did with zfs at sunacle.that zfs was bug free and that's why you have backups encase shit that would not happen with their bug free code happens in the first place :) 

> Not good for users, not good for the Fedora developers who support storage
> and file systems.

Depends on which user base you are referring to...

> Let's just keep the default in place and stay focused on the btrfs work.

We would not even be having the "what's the default" discussion if the so called designers of the newUI would simply have left the option to unhash lvm or hash lvm which is equally hard/easy for both parties to do, in the newUI

And this change was well know as Jesse stated in August on the Anaconda developers list... 

"The current default in the gui is to not use LVM, and to default to ext4. btrfs is not yet ready to be the default, and LVM is too complex of a setup for a large portion of our users.

The default autopartitioning scheme can include a broken out /home/ directory 
if there is sufficient disk space available, and a broken out /boot and swap.

It would be a good idea to review the autopart expectations in the QA tests and update them for code reality."

It's the *time* and introduction of this change it'a past freeze so the freeze needs to be lifted and the release slipped another week or two to allow us in QA to properly test it...

*cough* 
"The default autopartitioning scheme can include a broken out /home/ directory 
if there is sufficient disk space available, and a broken out /boot and swap."
*cough*

(In reply to comment #40)
> One separate note.
> 
> Converting in place from ext4 to btrfs OR in place from ext3 to ext4 is a
> *really* bad and silly thing to do for a ton of reasons.
> 
> It kind of appears to work, no one tests it, and when you report an upstream
> bug about it, you won't get support (more chuckles).
> 
> It was designed as a toy to let you play on a file system that you don't
> care about, not the preferred way to install a btrfs file system (or move to
> ext4 from ext3).
> 
> thanks!

I'm fine with not supporting it at all it's just a less mess we have to deal with in QA

Responding to the key points above, the non-LVM default which was done as part of a UI redesign was not broken out, called out in forums that the storage and file systems people follow and does directly break other Fedora features.

The change - which remove features from our users - was noticed in beta and should be reverted back to the F17 default.

This is not a "change" to a new stack, this is an explicit recognition that despite some individuals personal desire not to use LVM, this is the way it has been supported for years.

A major change in the storage and file system configuration needs to be called out explicitly and reviewed by all of the upstream/fedora communities impacted. This was not done in this case and we will inflict harm on our users.

Also note about forums, like most of the file system developers who support Fedora, I subscribe to *many* upstream lists, but not the anaconda devel list (ext4, nfs, btrfs, xfs, lkml, device mapper lists, IDE, SCSI, etc). 

We as a community need to figure out how to pull together a bit better on this kind of change. We have been getting a bit better (Linux Plumbers had anaconda developers, fs & storage developers and developers from other distros) spend a good chunk of time in late August discussing collaboration between SSM, libstoragemgt, LVM, anaconda, Suse Snapper, etc).

In effect, not reversing the change as this point in the release is actually making a change that did not get correct review by all of the parties involved in supported our users.

No release slip is needed - Adam and the QA team have been testing LVM. This is a simple tweak of a default, not a new, untested stack.

As for letting people use things that the developers proclaim is not supported, I am not sure what you mean. We definitely should not rely on in place conversion, that will burn the user base and I know that I don't want to pick up the pieces (and it is me and the other file system developers who have to support that).

As to Red Hat paid upstream developers not being Fedora, that is a silly comment.

We pay them to work on upstream and Fedora and RHEL. 

This default hurts Fedora users and is wrong. 

We would not do that to RHEL users either, but that is not the point of contention here.

(In reply to comment #43)
> No release slip is needed - Adam and the QA team have been testing LVM. This
> is a simple tweak of a default, not a new, untested stack.

Excuse me I'm part of the QA team and I can tell you here and now and I'm sure Adam will confirm it if you need confirmation from your coworker that we ( as in Fedora QA ) have not been explicitly been testing Autopart with LVM this whole time simply because of the fact that the newUI never had the opinion to *chose* to do it like the old UI did. Yes there have been few reporters that tested lvm and filed bugs but it was not something we explicitly targeted in QA and custom partitioning has been more or less broken this whole time... 

And what Jesse mentions warrants some concern in that regard...

We have to test lvm partitioning against the whole alpha and beta critera including upgrading with it and review bugs ( if any ) against those criteria and to do so we need time so I dont think not lifting the freeze  and slipping a week here is an option. 

Adam creating an update image and test if it works most likely in a vm is hardly what I call testing...

dam told us directly that he had been testing LVM.

I don't see how Jesse's suggestions that you quoted have bearing here - users don't have to do anything that they have not done in previous releases, unlike what the current default would foist on them which is a configuration that only power Fedora users have used (ext4 on raw partitions) since it required you to not use our existing defaults.

And it breaks other approved Fedora feature page projects as I stated earlier.


Switching away from a working, tested and documented stack without providing new documentation or getting sign off from the Fedora teams that need to support the users is not a good idea.

As I stated, leaving this is in effect a change that was not communicated and F18 users will be the worse for it.

There is - as far as I know - no QA plan that tests post install use of ext4 on raw partitions. 

Have you tested shrinking one partition and growing another without LVM? Migrating your file systems to a new, larger disk (which you can do with pluggable external storage today using documented, proven techniques)?  

Have you tested Storage System Manager without LVM?

Despite the assertion that LVM is too complicated (which runs counter to the fact that people use it now and have used it as the default for years), there has been no worked out plan to make sure that F18 works well *post install* without LVM.

That is where most of the life span of a box is.

Making this change to a non-LVM setup should and must be a formally planned "Feature Page" and needs proper planning.

Not just flip the switch and let others pick up the pieces.

One more higher level note.

Traditional file system - ext4, ext2, ext3, xfs - are designed to do just the file system bits. By intentional design, we don't do RAID, snapshots, etc in the file system code.

To get all of that functionality and full featured, modern IO stack, you need to add in those bits below the file system (lvm bits, multipathing, RAID, etc).

btrfs took a different tack - it rolls in those traditional block functions to provide ease of use.

What we get when we jump back to a raw partitions is a crippled IO stack that lacks features that all competing OS'es give their desktop users out of the box (windows, OS/X, etc).

We have expended a lot of community work to improve the usability of the stack and add clear value to normal users on LVM/ext4 (or xfs, etc).

Let's not please throw away all of this work in Fedora and roll back to the stone ages of storage one release before going to a default of btrfs where we can let users that don't want LVM to replace our existing, full featured stack with something that approached a full featured stack.

(In reply to comment #47)
> dam told us directly that he had been testing LVM.
> 
> I don't see how Jesse's suggestions that you quoted have bearing here -
> users don't have to do anything that they have not done in previous
> releases, unlike what the current default would foist on them which is a
> configuration that only power Fedora users have used (ext4 on raw
> partitions) since it required you to not use our existing defaults.
> 
> And it breaks other approved Fedora feature page projects as I stated
> earlier.

Can you elaborate on exactly how it is breaking other approved Features? 

> 
> Switching away from a working, tested and documented stack without providing
> new documentation or getting sign off from the Fedora teams that need to
> support the users is not a good idea.
> 
> As I stated, leaving this is in effect a change that was not communicated
> and F18 users will be the worse for it.

Sorry to say but the storage team was not the only team that was kept in the dark about changes in Anaconda so you have to suck it up like the rest of teams have had to do when they have detect missing but expect functionality in the installer then spent days rewriting criteria obsoleting test cases and what not or take it up with FESCO. 

> There is - as far as I know - no QA plan that tests post install use of ext4
> on raw partitions. 
> 
> Have you tested shrinking one partition and growing another without LVM?
> Migrating your file systems to a new, larger disk (which you can do with
> pluggable external storage today using documented, proven techniques)?  
> 
> Have you tested Storage System Manager without LVM?
> 
> Despite the assertion that LVM is too complicated (which runs counter to the
> fact that people use it now and have used it as the default for years),
> there has been no worked out plan to make sure that F18 works well *post
> install* without LVM.
> 
> That is where most of the life span of a box is.
> 
> Making this change to a non-LVM setup should and must be a formally planned
> "Feature Page" and needs proper planning.
> 
> Not just flip the switch and let others pick up the pieces.

Flip the switch I'm just saying that if we do we ( QA community not just Adam ) have to test lvm partitioning against the whole alpha and beta critera including upgrading with it and review bugs ( if any ) against those criteria. 

And honestly I cant figure out why you are against that since we would be *ensuring* to the best of our ability that lvm functionality was good enough for our end user base. 

Another week slippage is not an issue since we have been sliding down hill this whole release cycle, sliding few meters more hardly matters now and if fesco had not decided to "freeze" ( which was done against QA recommendations since we had not been able to test the upgrade tool or even had it present repository  at that time ) we would have slipped anyway another week...

We have posted above the various bits that break without LVM.

Beta is not a time to "suck it up" and do a disservice to our users, it is a time to fix things that have been done incorrectly.

This is a big deal and Fesco will need to grant the exception and/or slip the date if need be.

That is what beta and QA are about - finding and fixing bugs before they get inflicted on the broad user base not just ticking off boxes in a schedule :)

(In reply to comment #50)
> We have posted above the various bits that break without LVM.

No you have not, not in direct relation with those features. 

All you have said is that users wont be able to use those features which only affects those users that had any intent on using it and assuming that all users do, want or will is just absurd for this feature any other feature for that matter.

> Beta is not a time to "suck it up" and do a disservice to our users, it is a
> time to fix things that have been done incorrectly.

Changes like these should happen before change deadline,100% feature completion and "freeze" 

> This is a big deal and Fesco will need to grant the exception and/or slip
> the date if need be.

If they grant exception they will need to slip it we need to test this...
 
> That is what beta and QA are about - finding and fixing bugs before they get
> inflicted on the broad user base not just ticking off boxes in a schedule :)

Yes we are the pest control for the distribution ;)

(In reply to comment #12)
> Adam asked me to add this to the bug so:
> 
>  don't have current numbers but it adds a synchronisation point (i.e
> breaks a fully parallel bootup) let me just quote this
> (http://0pointer.de/blog/projects/blame-game.html):
> 
> "Let's have a closer look at the worst offender on this boot: a
> service by the name of udev-settle.service.
<snip, followed by a recommendation to disable it>

AFAIK we always have the service enabled whether LVM is or isn't used, so this aspect (and all of the associated subthread about alternatives) seems to make absolutely no difference right now.

(Longer term it may make a difference, but is anyone proposing to rework these things after the beta freeze?)

Here are specific bits that leaving LVM disabled breaks:

* SSM - http://fedoraproject.org/wiki/Features/SystemStorageManager
* all documentation about how to manage your file and storage post install
* ability to resize your file systems (using an existing tool, system-config-lvm) and enable snapshots or other advanced features
* ability to migrate using the documented tools from one disk to another
* ability to support multi-pathed storage (needed not just in servers, but also in other uses cases)

Flipping the default back to our historic settings avoids this mess.

Disabling and crippling the stack in F18 is a regression.

(In reply to comment #53)
> Here are specific bits that leaving LVM disabled breaks:
> 
> * SSM - http://fedoraproject.org/wiki/Features/SystemStorageManager
> * all documentation about how to manage your file and storage post install
> * ability to resize your file systems (using an existing tool,
> system-config-lvm) and enable snapshots or other advanced features
> * ability to migrate using the documented tools from one disk to another
> * ability to support multi-pathed storage (needed not just in servers, but
> also in other uses cases)


So I get you right... 

Are you claiming that users no longer can manually can create lvm partitions either on the local disk, multi-pathing wont work or an external and take advantage of this feature if we dont default to lvm *at install time*? 

Or are you simply saying user wont be able to take advantage of the above unless they manually create lvm partitions, setup multipathing etc before doing so?

Clearly, we are talking about what happens to users with the current (LVM disabled) default.

For clarity, if the default install on a laptop or desktop box consumes all space in our default installation, you will not be able to flip those raw partitions + ext4 over to an LVM + ext4 file system in place.

(In reply to comment #52)
> (In reply to comment #12)
> > Adam asked me to add this to the bug so:
> > 
> >  don't have current numbers but it adds a synchronisation point (i.e
> > breaks a fully parallel bootup) let me just quote this
> > (http://0pointer.de/blog/projects/blame-game.html):
> > 
> > "Let's have a closer look at the worst offender on this boot: a
> > service by the name of udev-settle.service.
> <snip, followed by a recommendation to disable it>
> 
> AFAIK we always have the service enabled whether LVM is or isn't used, so
> this aspect (and all of the associated subthread about alternatives) seems
> to make absolutely no difference right now.

The difference is you can safely disable it (along with the other crap we enable by default that hardly anyone uses) ... while with lvm you *cannot* disable it and expecting your system to boot.

(In reply to comment #55)
> Clearly, we are talking about what happens to users with the current (LVM
> disabled) default.
> 
> For clarity, if the default install on a laptop or desktop box consumes all
> space in our default installation, you will not be able to flip those raw
> partitions + ext4 over to an LVM + ext4 file system in place.

Nor will you be able to do the opposite if you start with lvm and want to migrate to, say, btrfs.

This is about the default installation options, not how to migrate to btrfs from ext4.

No sane user should *ever* use the migrate in place from ext4 to btrfs, you should lay down a new file system with its default and tested options.

That said, there is nothing to prevent you from with running btrfs on top of device mapper or LVM if you want to do that (including doing that ill advised convert in place migration).

(In reply to comment #58)
> This is about the default installation options, not how to migrate to btrfs
> from ext4.

It's about migrating from lvm to plain partitions (a good use-case for which is groundwork for migrating to btrfs).

> 
> No sane user should *ever* use the migrate in place from ext4 to btrfs, you
> should lay down a new file system with its default and tested options.

That's not what I was talking about.

> 
> That said, there is nothing to prevent you from with running btrfs on top of
> device mapper or LVM if you want to do that (including doing that ill
> advised convert in place migration).

Of course not, but you're assuming people want both.

The point is you're at least as locked-in when you start with lvm as you are with plain partitions.

Not sure what you are trying to point out - you always need to reinstall when you change your file system type:

If you want to migrate in place from ext4+LVM to btrfs on raw partitions, you need to reinstall your box.

If you want to migrate from ext4 to btrfs on raw partitions, you also need to reinstall your box.

If you want to migrate from ext3 to ext4 (with or without LVM), you need to reinstall.

If you want to migrate from btrfs on lvm to raw btrfs, you need to reinstall.

(In reply to comment #35)
> Matthias - there's a use_lvmetad option, IIRC. That being said, it's likely
> somewhat tangential to this issue.

Agreed: bug 870499

(In reply to comment #60)
> Not sure what you are trying to point out - you always need to reinstall
> when you change your file system type:

I skipped a comment or two and may have interpreted the second paragraph of comment 55 out of context. Without context it looked like you were trying to (falsely) imply that it's harder to migrate to lvm in-place than it is to migrate away from it (the lock-in factor).

This jumps out at me because of the many other half-truths being used as arguments, like the notion that you can't resize filesystems without lvm. Anyway, carry on.

This entire thread is about the default install for non-power users.

Migrating in place is not a good idea, without or without LVM, but that is entirely not the context of the thread.

Resizing a file system is a supported and documented option that you can do with a GUI using LVM.

Can we do the same in a useful way on raw partitions?

How would you get a naive user to give 10GB more to one file system without LVM?

I don't really understand why there is such a rush to jump backwards in functionality.

Keep in mind that once anaconda has done its job, the user (including non-power users) have normal, operational tasks that our user base relies on LVM for.

Let's not "change the world" twice in back to back releases without proper coordination with all of the concerned parties (storage, file system and anaconda people) working together to make sure it is done properly.

For F19, it would be great to have you as the anaconda storage lead work more closely with the file and storage teams so we can iron out these critical bits before beta.

Johann: I wouldn't say we *definitely* need a slip to do the testing. We have to do the whole test matrix against the RC anyway, and we always get through that. If the change doesn't actually cause any problems, I don't think we'd have any trouble getting through all the testing. If it _did_ cause a problem, we might be more prone to a slip.

On upgrades - we haven't done any successful upgrade tests yet anyway and this change doesn't really affect upgrades as it's a change to F18, not F17; we don't fiddle with the filesystems on upgrade (and upgrades aren't even going to go through anaconda any more).

So if we flip the switch we have to test upgrade of an ext4-on-LVM F17 system to F18 using the new upgrade tool...and if we do flip the switch, we have to test upgrade of an ext4-on-LVM F17 system to F18 using the new upgrade tool. No change there. The codepaths that would be changed for this bug would never be touched on upgrade.

Given the minor nature of the change - 'What Could Possibly Go Wrong' theory aside - I'm still more worried about the upgrade tool causing a slip than this causing a slip. This at least is known code we've been using for several releases. The upgrader is completely new stuff.

On a couple of specific points that were raised - Ric, Johann is pretty much right about the state of existing testing, sorry if I was unclear about that. We've tested *LVM installs* of F18 already, to some extent, but as Johann says, up till now you've had to go through custom partitioning to do an LVM install, and that's a rather different workflow from using the automatic partitioner. Johann is correct to say that until yesterday no-one had done any testing of the autopart algorithm using LVM in F18, because it wasn't possible; there was no way to tell the autopart code to use LVM.

Johann, I don't read Jesse's statement as any kind of change from previous releases. For several releases autopart has been creating /home when the target disk is of a certain size, and not creating it when the target disk is of a smaller size - whether using LVM or not. I think Jesse was just explaining existing behaviour to someone in that quote.

Ric: you can resize raw partitions pretty easily, and graphically, by booting a Fedora live image and running the GNOME Disks tool. I think you'd be more accurate in saying that it's somewhat easier, safer and more flexible to resize partitions using LVM (e.g. you can do it live and you can resize partitions from the front, which you can't do with raw ext4) than to keep saying 'you can't resize partitions without LVM', which isn't really true.

Fair enough - you can use a GUI to shrink your file systems.

What you cannot do is to easily reuse that freed up space though.

In the default install, if you have one file system for boot, one for system and one for home, you can certainly shrink using GNOME disk util.

But you won't be able to add space back easily to the other file system (since the space freed up is not contiguous, you need LVM to give you that capability).

File systems are not multi-partition aware (again outside of btrfs).

(In reply to comment #53)
> Here are specific bits that leaving LVM disabled breaks:

It occurs to me I have not yet stated that I don't have a strong preference for either default, provided the installer offers an easy way to opt out of it.

Having said that...

> * all documentation about how to manage your file and storage post install

All RH/Fedora docs? Okay. All docs? Nowhere near the truth.

> * ability to resize your file systems (using an existing tool,
> system-config-lvm) and enable snapshots or other advanced features

resize: gnome-disk-utility, gparted

> * ability to migrate using the documented tools from one disk to another

dump/restore?, dd?, rsync?

Oh -- documented by RH/Fedora. Fair enough.


The fact remains that lvm can do all of this stuff and offers a somewhat unified front, which has advantages.

I am talking about Fedora and RH docs (which are the most relevant to our Fedora user base I assume).

Resize is not sufficient unless you can also reuse the space in the common case of wanting to shift space from one partition to the next. You can shrink two adjacent partitions to be smaller with the tools you mention, but you cannot add in say 10GB that you shrunk from sda2 to the end of a file system on sda3.

dump/restore are not really actively maintained for ext4. We compile test them, but last time I was at the ext4 developers mini-summit *none* of us had used either tool for years. Probably worth deprecating.  (XFS is different here).

dd is certainly not for end users and won't work on a live, mounted file system. Your file system state at the block level is not consistent (needs to be unmounted for that copy).

rsync or tar would both work, but they both "copy" files, they change file system state (neither is a block by block image of the original).

Having a documented and well tested tool chain for these bits is critical.  

We will have a ton of work to get btrfs ready for the F19, this is just going to be a distraction for us.

Just for clarity, in https://bugzilla.redhat.com/show_bug.cgi?id=870207#c68, I meant to say that rsync and tar do not do block device level "block by block" copies so the inode layout and so on will change from the source.

https://fedorahosted.org/fesco/ticket/964

(In reply to comment #69)
> Just for clarity, in https://bugzilla.redhat.com/show_bug.cgi?id=870207#c68,
> I meant to say that rsync and tar do not do block device level "block by
> block" copies so the inode layout and so on will change from the source.

There is also fsarchiver which does not do a block by block copy either but does preserve the options the filesystem has been created with (which is all what matters really the block layout on disk is not really interesting from a users pov).

For anyone who wants to help test the change - as well as the updates.img I already posted for anaconda 18.19 (Beta TC6), which is at http://adamwill.fedorapeople.org/updates-lvm.img , I've built an updates.img for anaconda 18.21, which is in smoke12 and may be in TC7:

http://adamwill.fedorapeople.org/updates-lvm-1821.img

so you can test against the two latest builds of anaconda now.

(In reply to comment #72)
> For anyone who wants to help test the change
...
Thanks Adam for this constructive approach!

If it helps anything, I tried TC6 and updates-lvm.img from comment #72
(only minimal install, I do not care about about desktop now).

1) Default LVM based installation works, it seems to look like the system is working as in F17.
2) The default "encrypted system" option (LVM over LUKS) seems to be working as well.

The only bug I see for default partitioning (not blocker now) is that it always use default VG name - bug #870684

But I have to mention that manual LVM configuration GUI is still incredibly confusing, more precisely, I was not able to set up LVM or LUKS options there at all... Whatever, manual kickstart should work. I am not using LVM on testing machines anyway 8-)
I'll try TC7 when it is out and report it later though.

Created attachment 635247 [details]
calc file, test matrix

Adam,

I tested with the updates-lvm.img for F18b TC6, a guest with only one disk.
What i found (using 'automatic partitioning') is:

1* anaconda uses CODE 0700 for /boot when using GPT disklabel, but it should use 8300. (not a big problem)

2* GPT disks are reverted to MSDOS if there are no partitions created.
3* GPT disks are reverted to MSDOS if one reclaims all partitions. (ACTION=DELETE)

When using 'gpt' boot parameter:
4* MSDOS disks are converted to GPT if there are no partitions.
5* MSDOS disks are converted to GPT if one reclaims all partitions. (ACTION=DELETE)

Can you confirm these results?

I attached a cacl spreadsheet.

Reartes: does any of that have anything at all to do with LVM? On the face of it I don't see how. Did you check if the behaviour with LVM is at all different from that without it?

Sorry, i should have added that LVM worked (it says so inside the cacl file, it seems i forgot to put it outside too.).

So far as this bug goes we are only concerned with cases where the behaviour with the updates.img differs from the behaviour without it in what is clearly a bad way. For the record, 2, 3, 4 and 5 are all the behaviour I'd expect. I don't know about 1 but it seems out of the scope of this bug report.

Number of fesco members 9 

tmraz +1 comment 9

" I'm curious the outcome of the smoketesting asked for last week with LVM enabled? Any big problems found?
If not, I am for changing it back to lvm by default for f18. "

No big problems where discovered during testing which means kevin +1  comment 12

limb +1 comment 13

mitr +1 comment 15

mmaslano +1 comment 15

Total number of +5

Total number to reach majority = 5 which means majority achieved to go back to LVM-by-default for F18... 

This bug can be closed now

Well it can't be closed, because the change hasn't been made.

For blocker/NTH accounting purposes: we will count FESCo's decision as mandating that this bug be accepted as NTH, as we effectively escalated the NTH decision to FESCo. I'm therefore marking the bug as AcceptedNTH. David, please apply the LVM-by-default patch to anaconda git and pull it into the next build (18.22?) Thanks.

Note that at this point we are more or less operating under the assumption that we will be slipping by another week. The current blocker bug load and the state of fedup make this extremely likely.

This wasn't in 18.22. Was this an oversight or is the change not baked yet?

anaconda-18.23-1.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/anaconda-18.23-1.fc18

Package anaconda-18.23-1.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing anaconda-18.23-1.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-17509/anaconda-18.23-1.fc18
then log in and leave karma (feedback).

anaconda-18.24-1.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/anaconda-18.24-1.fc18

Package anaconda-18.24-1.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing anaconda-18.24-1.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-17543/anaconda-18.24-1.fc18
then log in and leave karma (feedback).

anaconda-18.25-1.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/anaconda-18.25-1.fc18

anaconda-18.26-1.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/anaconda-18.26-1.fc18

18.26 went stable. Closing. (Bodhi closing of bugs when updates go stable is currently broken).