Bug 737261

Summary: libguestfs grub-install API needs grub1
Product: [Fedora] Fedora Reporter: Tom Horsley <horsley1953>
Component: libguestfsAssignee: Richard W.M. Jones <rjones>
Status: CLOSED EOL QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 19CC: awilliam, dennis, eblake, jfeeney, jonathan, kchamart, lkundrak, mads, mbooth, metherid, mishu, pjones, ratmice, ricardo.arguello, rjones, robatino, vanmeeuwen+fedora, virt-maint
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-02-17 08:52:28 EST Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:

Description Tom Horsley 2011-09-10 07:51:08 EDT
Description of problem:

I try an f16 x86_64 install from the TC2 DVD iso
image (using hard disk install technique).

I right click on lots of things and say "select
all optional packages"

I get to the end of package selection and clicking
Next gives me a popup warning that says something
like:

"Some of the packages you have selected for install
are missing dependencies. Yadda, yadda, yadda..."

Clicking the "Details" button, however, does not show
a missing dependency, instead it shows a conflict:

grub conflicts with 1:grub2-1.99-5.fc16.x86_64

Clicking the "Back" button (which is one of the options)
to try to fix this is utterly useless. There is no
search option in anaconda, and hunting and pecking
through lots of optional package lists does not
find "grub" anywhere in order to deselect it.

Finally clicked the "Continue" button to install
anyway even with problems.

Version-Release number of selected component (if applicable):

This was the Sep 9 F16 Beta TC2 DVD iso, not sure what specific version
numbers are on that DVD.

How reproducible:
Only tried once

Steps to Reproduce:
1.see above
2.
3.
  
Actual results:
Impossible to resolve warning.

Expected results:
No warnings.

Additional info:
Comment 1 Andre Robatino 2011-09-10 08:07:30 EDT
I was told by mether on #fedora-qa that the conflict is intentional:

<mether> robatino, yes,  that is intentional
<mether> fedpkg clone grub2 and look at the spec.  it has a explicit conflicts
Comment 2 Tom Horsley 2011-09-10 08:41:40 EDT
So the question is, why are they both on the DVD and both apparently
auto selected for install? At least I couldn't find either of them
in any of the lists of optional packages I could manually select
or unselect.
Comment 3 Tom Horsley 2011-09-10 09:36:29 EDT
Some other random package dragged in by one of my "select all optional
packages" must have grub as a dependency. I just checked a default
f16 install from the same DVD iso, and I do not get the conflict when
I just accept all the defaults. What could possibly have grub as
a dependency, I have no idea.
Comment 4 Adam Williamson 2011-09-10 14:45:35 EDT
[root@adam adamw]# repoquery --whatrequires grub
firstaidkit-plugin-grub-0:0.3.1-1.fc16.x86_64
libguestfs-1:1.12.4-1.fc16.i686
libguestfs-1:1.12.4-1.fc16.x86_64
libguestfs-1:1.12.6-1.fc16.i686
libguestfs-1:1.12.6-1.fc16.x86_64
ovirt-node-0:2.0.0-1.fc16.noarch
ovirt-node-0:2.0.1-4.fc16.noarch

I'm guessing libguestfs.
Comment 5 Adam Williamson 2011-09-10 14:48:35 EDT
so, there's one case which worries me here, which we ought to test. what happens when the installer uses grub on purpose - EFI install? grub2 is 'default' in the Core package group, so will anaconda wind up trying to install both grub and grub2 in that scenario? if so, that's a problem.
Comment 6 Tom Horsley 2011-09-10 17:16:59 EDT
Installing libguestfs is not a strange or uncommon operation either. If
people are going to get the conflict every time they select the
virtualization group to install during anaconda, they are going to be
large numbers of mightily confused folks out there :-).
Comment 7 Andre Robatino 2011-09-10 21:44:24 EDT
If I try to install libguestfs in my F16 guest, which is currently using grub2, it ends up trying to pull in grub and fails due to the conflict. On the other hand, "yum groupinstall Virtualization" works.
Comment 8 Andre Robatino 2011-09-11 22:24:29 EDT
During a GUI install, under package customization, Base System/Virtualization, if I choose any of the packages guestfish, guestfs-browser, libguestfs-tools, or python-libguestfs, I get the grub/grub2 conflict error. If I just check the Virtualization box, none of these are default choices and the install proceeds. I don't see an explicit grub2 choice under any of the sections, so no obvious way to deselect it.
Comment 9 Mads Kiilerich 2011-09-13 10:39:21 EDT
FWIW, IMHO: There is no good technical reason that grub and grub2 should conflict. The sysadmin obviously has to choose which one to use, and having an unused boot loader package installed might be confusing, but that shouldn't be solved with a conflict.

(grub and grub2-efi conflicts because they both provide a file none of them IMHO should provide.)
Comment 10 Richard W.M. Jones 2011-09-15 09:51:34 EDT
libguestfs really needs grub (and in future, grub & grub2 in
parallel) since it may need to invoke either grub1 or grub2 on
guests.  For example if you had F15 and F16 guests installed
together.

So it should be possible to parallel-install both versions.

From a quick look at the spec files, it doesn't look to me as
if there are any real file conflicts.  It appears to be just an
artificial "Conflicts: grub2" that has been added (only to the
F16 branch for some reason).

Please remove this artificial "Conflicts" as it is causing
a lot of trouble for our users.
Comment 11 Rahul Sundaram 2011-09-15 10:06:16 EDT
FWIW,  Fedora recommends not using conflicts as much as possible ->  https://fedoraproject.org/wiki/Packaging:Conflicts
Comment 12 Mads Kiilerich 2011-09-15 10:11:29 EDT
(In reply to comment #10)
> From a quick look at the spec files, it doesn't look to me as
> if there are any real file conflicts.  It appears to be just an
> artificial "Conflicts: grub2" that has been added (only to the
> F16 branch for some reason).

That was done as a workaround to fix bug 725185.
Comment 13 Matthew Garrett 2011-09-15 10:22:08 EDT
Having two bootloaders installed is not a supported configuration. Nothing should be depending upon a specific bootloader.
Comment 14 Eric Blake 2011-09-15 13:10:23 EDT
Having two bootloaders installed is mandatory for virtualization support.  I agree with comment #10, and this is preventing me from running some of the tests for Fedora Virt Test Day:
https://fedoraproject.org/wiki/Test_Day:2011-09-15_Virtualization
Comment 15 Matthew Garrett 2011-09-15 13:20:16 EDT
It's a bug for libguestfs to assume that it can run the host's bootloader against an arbitrary guest, and it's a bug for it to assume that it can depend on grub being present. I'm sorry that this issue wasn't picked up until the virt test day - I appreciate that this makes running the tests significantly harder.
Comment 16 Eric Blake 2011-09-15 13:33:44 EDT
libguestfs isn't assuming that it can run the host's bootloader against the guest, so much as it is probing the guest's bootloader, then reproducing that by use of the host.  There is nothing inherently wrong with having _both_ grub and grub2 in the host, both as requirements of libguestfs, so that libguestfs can then use the proper bootloader required by the particular guest image it will be manipulating.  In order to manipulate a guest that uses grub2, libguestsfs must have grub2 installed on the host.  In order to manipulate a guest that uses grub, libguestfs must have grub installed on the host.  Libguestfs doesn't care which of the two bootloaders the host uses, but since both guests may be present in parallel, libguestfs really does require that both bootloaders be present on the host.

You are trying to argue that since an F16 host with a grub2 bootloader is not using grub, then grub should not be installed.  But you are overlooking the fact that there are still valid use cases for having grub present, even if the host does not use it, because virt tools can indeed use grub for guests.
Comment 17 Adam Williamson 2011-09-15 13:36:18 EDT
when you say 'the host's bootloader'...it doesn't seem so straightforward as that. To me 'the host's bootloader' is what's in the MBR or root partition header, not what's installed as a distro package. Those are...potential bootloaders, I guess.

"a supported configuration" is a slippery phrase, as well. What exactly do you mean by that? It's not as if anyone has a support contract with Fedora. Are you talking about upstream? Or just 'I don't think people should do that'?

Is there an actual technical reason why the two must never be installed side-by-side to justify the conflicts? Or is it just a case of 'well, we don't think anyone should do that, and it works around a bug, so let's put it in'?

there's some interesting discussion in https://bugzilla.redhat.com/show_bug.cgi?id=725185 , anyway.
Comment 18 Matthew Garrett 2011-09-15 13:45:35 EDT
You can't assume that F16's grub legacy will work with an arbitrary grub-legacy based guest. You can pretty much guarantee that F16's grub 2 *won't* work with an arbitrary grub 2 based guest. The only bootloader that you can reliably run against a guest is the one that's installed in the guest. So when you say "libguestfs really does require that both bootloaders be present on the host", what you actually mean is "libguestfs really does require that every version of grub that Fedora, Ubuntu and any other libguestfs-supported distribution have ever used be present on the host", and that's clearly not possible.
Comment 19 Eric Blake 2011-09-15 14:08:18 EDT
http://koji.fedoraproject.org/koji/taskinfo?taskID=3353628 removes the Conflicts tag, at least for the purposes of testing if any other problems exist with a parallel install
Comment 20 Eric Blake 2011-09-15 14:12:12 EDT
Correction,(In reply to comment #19)
> http://koji.fedoraproject.org/koji/taskinfo?taskID=3353628 removes the
> Conflicts tag, at least for the purposes of testing if any other problems exist
> with a parallel install

Correction - that's a libguestfs build that removes the Requires:grub, to avoid the conflicts problem while sorting out what libguestfs really needs to support arbitrary guest grub.
Comment 21 Tom Horsley 2011-09-15 14:21:12 EDT
Pretty much 99% of my libguestfs usage is on Windows XP guests, and I'm pretty
sure it doesn't use grub or grub2, yet somehow libguestfs deals with XP.
Maybe it should deal with linux the same way (whatever that way happens to
be :-).
Comment 22 Richard W.M. Jones 2011-09-15 14:29:08 EDT
There are various conflated issues here.

libguestfs does a huge amount of things, and only 1 (out of ~300)
APIs depends on running /sbin/grub-install.
http://libguestfs.org/guestfs.3.html#guestfs_grub_install
So not having grub-install does not "break" libguestfs.  It just
breaks one very small feature of libguestfs, one that many people
may never use (see comment 21).

Yes, we do run a different grub1 from what is in the guest.
However we do it in very controlled circumstances -- basically
when we know that it will work, eg. in virt-v2v where we tightly
control what guests we support.

I originally wanted to use grub2 on general guests (eg. Ubuntu).
However it does indeed look like this will be far more problematic
and error-prone than I thought it would be.

However this does point to a big problem.  A big problem in grub.
Which is that it should be possible to reinstall a boot loader
without needing to run guest code, which is inherently a security
nightmare.  (Compare for example if you could not run "mkfs"
except by using the "mkfs" from the guest -- that would be a
ridiculous situation, yet it's the same for grub).
Comment 23 Matthew Garrett 2011-09-15 14:41:36 EDT
If you run mkfs and then boot a sufficiently old guest, the guest won't be able to read the filesystem. It's different for grub only in that it doesn't guarantee forward *or* backward compatibility. The expectation is that the installed version is the running version, and fixing that is (a) difficult and (b) not something that upstream seems worried about.
Comment 24 Richard W.M. Jones 2011-09-15 14:51:18 EDT
(In reply to comment #23)
> If you run mkfs and then boot a sufficiently old guest, the guest won't be able
> to read the filesystem.

Yeah I knew I was walking into that one as I was typing it :-)
Indeed there are kernel compatibility issues with this, but by
use of mke2fs options you can resolve all of them (AFAIK) and
create a fully backwards compatible filesystem.

The reverse of this case is also the reason why virt-v2v requires
that the host be at least as new as the guest.  eg. virt-v2v on
Fedora X refuses to deal with Fedora > X, and the same on RHEL.

> It's different for grub only in that it doesn't
> guarantee forward *or* backward compatibility. The expectation is that the
> installed version is the running version, and fixing that is (a) difficult and
> (b) not something that upstream seems worried about.

It's still a bug in grub even if they aren't thinking
about it.
Comment 25 Matthew Garrett 2011-09-15 14:58:25 EDT
It's expected behaviour. You could describe it as undesirable behaviour, but I suspect that upstream's perspective is that it's not a use case they intended to support and so isn't a bug.
Comment 26 Tom Horsley 2011-09-15 15:04:11 EDT
I gotta say it would never occur to me that a different grub than the
one on the guest might work. If confronted with fixing a guest
boot problem, I would probably never think about guestfs, I'd probably boot
the guest from a rescue CD image and do a chroot and grub-install
from inside the guest just like I do on real hardware when I need to
fix a busted grub.
Comment 27 Eric Blake 2011-09-15 15:12:57 EDT
Yet that's precisely what virt-resize has to do when resizing a virt image with a grub MBR - if it detects grub was installed in the old image, then it has to reinstall grub in the new image to point to the new partition layout, all without the aid of running any guest code.
Comment 28 Richard W.M. Jones 2011-09-15 15:19:38 EDT
(In reply to comment #26)
> I gotta say it would never occur to me that a different grub than the
> one on the guest might work. If confronted with fixing a guest
> boot problem, I would probably never think about guestfs, I'd probably boot
> the guest from a rescue CD image and do a chroot and grub-install
> from inside the guest just like I do on real hardware when I need to
> fix a busted grub.

I trust you'd use "virt-rescue" :-)

With libguestfs we're trying to do the hard stuff here.  Suppose
you had to fix 1000 guests?  Or you wanted to write a self-service
web interface to fix guests (ie. no root permitted)?  Ad hoc
manual rescuing is not acceptable in all cases.

(In reply to comment #27)
> Yet that's precisely what virt-resize has to do when resizing a virt image with
> a grub MBR - if it detects grub was installed in the old image, then it has to
> reinstall grub in the new image to point to the new partition layout, all
> without the aid of running any guest code.

To be fair here, virt-resize attempts to preserve the original
bootloader.

virt-v2v used to use grub-install (for some very arcane reasons).
I believe in the latest version we now try to preserve the
original bootloader also.
Comment 29 Tom Horsley 2011-09-15 15:53:39 EDT
> With libguestfs we're trying to do the hard stuff here.  Suppose
> you had to fix 1000 guests?  Or you wanted to write a self-service
> web interface to fix guests (ie. no root permitted)?  Ad hoc
> manual rescuing is not acceptable in all cases.

Wouldn't necessarily have to be ad-hoc: If confronted with doing this
for 1000 guests, I'd seriously consider devising a custom rescue CD
image that does the necessary work automatically as soon as it boots
with a built in rc.local script or something.

Of course, I'd prefer to not have a job where the issue ever came up :-).
Comment 30 Mads Kiilerich 2011-09-15 16:41:37 EDT
(In reply to comment #18)
> You can't assume that F16's grub legacy will work with an arbitrary grub-legacy
> based guest. You can pretty much guarantee that F16's grub 2 *won't* work with
> an arbitrary grub 2 based guest. The only bootloader that you can reliably run
> against a guest is the one that's installed in the guest.

I thought one of the lessons from bug 735259 was that the boot loader package shall not touch the running configuration. A boot loader package is essentially a boot loader installer. That installer - when run by the sysadmin or anaconda - will (unfortunately) install files and sectors outside the control of the package manager, but that is how it has(?) to be in an rpm-based "don't ask" system.

So what does it mean that a guest is grub2 based? I assume it means that some grub2 installer has installed a self-contained and consistent set of sectors and files. Why does it matter where that installer came from or what grub version it was, as long as it does its job correctly and start the right kernel in the right environment with the right parameters?

Does it all boil down to the necessity of installing a boot loader that understands the format/version of the boot loader configuration file?
Comment 31 Matthew Garrett 2011-09-15 16:46:42 EDT
Grub 2 is modular and allows commands to be inserted at runtime. An Ubuntu configuration file may refer to commands that we don't build in the Fedora grub 2, for example. Attempting to load a module from the Ubuntu grub 2 build into the Fedora grub 2 could fail, since they can access various internal structures that may have changed.
Comment 32 Adam Williamson 2011-10-04 13:46:05 EDT
updating the summary as we've spun this bug off to be about the whole 'should libguestfs use the host's bootloader packages at all' debate, it's not about the DVD any more per se.
Comment 33 Tom Horsley 2011-10-04 18:06:10 EDT
Whatever the summary is, I can report that the same grub/grub2 conflicts
pops up in the just released F16 Beta DVD. It is gonna be confusing
lots more people if it is still there in the final f16.
Comment 34 Richard W.M. Jones 2011-10-05 05:47:49 EDT
Fix summary.
Comment 35 Fedora End Of Life 2013-04-03 13:22:36 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 19 development cycle.
Changing version to '19'.

(As we did not run this process for some time, it could affect also pre-Fedora 19 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 19 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora19
Comment 36 Fedora End Of Life 2015-01-09 11:46:40 EST
This message is a notice that Fedora 19 is now at end of life. Fedora 
has stopped maintaining and issuing updates for Fedora 19. It is 
Fedora's policy to close all bug reports from releases that are no 
longer maintained. Approximately 4 (four) weeks from now this bug will
be closed as EOL if it remains open with a Fedora 'version' of '19'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 19 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.
Comment 37 Fedora End Of Life 2015-02-17 08:52:28 EST
Fedora 19 changed to end-of-life (EOL) status on 2015-01-06. Fedora 19 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.