FWIW I tested rebooting in EC2 and it _does_ work. However, that's not at all surprising, as that's using PVM rather than HVM (and therefore, not using the boot loader).

Tentatively reassigning to cloud-utils, on the theory that growpart is to blame.

There's been at least one other person that ran into this issue with libvirt as well. We don't have cloud specific criteria for this, but I think the Beta Shutdown, Reboot, Logout Criteria fits here: "Shutting down, logging out and rebooting must work using standard console commands and the mechanisms offered..."

(In reply to Matthew Miller from comment #2)
> Tentatively reassigning to cloud-utils, on the theory that growpart is to
> blame.

Confirmed.

If I instruct cloud-init to not grow my partition then I can reboot as much as I want. I am using virt-install --import and using a local iso image as a source for cloud-init data. Here is the user-data I am passing in:

[dustymabe@localhost guests]$ cat my-user-data 
#cloud-config
password: passw0rd
chpasswd: { expire: False }
ssh_pwauth: True
growpart:
  mode: off


After booting an instance in this manner I then run growpart manually:

[root@localhost ~]# growpart /dev/sda 1                                                                                                                                                              
CHANGED: partition=1 start=2048 old: size=6144000 end=6146048 new: size=6286612,end=6288660

And... I can no longer reboot anymore.

NOTE: I also tried running fdisk manually and adding a 2nd partition instead of growing the first partition to make sure that any old change to the partition table didn't affect reboot. Making this did not inhibit me from rebooting.

This problem was not present in the F20 cloud images because those images used the syslinux MBR by default, which does not appear to be affected by this issue.

For both the syslinux-mbr and the original mbr, there is a change to the mbr after the first reboot:

$ diff mbr-pre-reboot mbr-post-reboot
28,29c28,29
< 00001b0: 0000 0000 0000 0000 0023 c818 0000 8020  .........#..... 
< 00001c0: 2100 8392 547e 0008 0000 00c0 5d00 0000  !...T~......]...
---
> 00001b0: 0000 0000 0000 0000 0023 c818 0000 8066  .........#.....f
> 00001c0: 0900 8392 d4ff 0008 0000 18f4 7f02 0000  ................

I don't know whether this represents (a) something writing where it shouldn't or (b) something that is expected but to which the non-syslinux mbr is sensitive.

Summarizing by request:

- The F21 cloud images have a boot loader that appears to be installed by parted.

- There is some change to the mbr as part of the intial boot/resize/reboot process.  I do not have enough mbr-fu to know what's going on here (is it expected? or a bug?)

- The parted-derived mbr is unable to boot the image after the initial boot/resize/reboot.

- Using a syslinux-derived mbr -- applied either to the original image or before reboot -- seems to avoid the problem.  There is still a change to the mbr, but it does not prevent a sucessful reboot.

It is not clear whether this problem is due to a bug in growpart, a bug in the parted-dervied mbr, or something else.  Using the syslinux mbr seems to avoid the problem.

Workarounds:

From inside a vm, after booting a cloud image:

    # dd if=/usr/share/syslinux/mbr.bin of=/dev/vda

To fix a qcow2 image on disk:

    # qemu-nbd -c /dev/nbd0 fedora-cloud-base.qcow2
    # dd if=/usr/share/syslinux/mbr.bin of=/dev/nbd0
    # qemu-nbd -d /dev/nbd0

To fix a raw image on disk:

    # dd if=/usr/share/syslinux/mbr.bin of=fedora-cloud-base.img conv=notrunc

One possibile workaround is to put

dd if=/usr/share/syslinux/mbr.bin of=/dev/vda

in the kickstart post script used when the images are generated.

Discussed at 2014-10-03 blocker review meeting: http://meetbot.fedoraproject.org/fedora-blocker-review/2014-10-03/f21-blocker-review.2014-10-03-15.58.log.txt . Accepted as a blocker per criterion "Shutting down, logging out and rebooting must work using standard console commands and the mechanisms offered..."

Verified again on TC3 base image on Openstack.

Growpart is not doing anything wrong AFAICT. In fact, disabling growroot and running the following already breaks the subsequent boot:
$ sfdisk -d /dev/vda > mbr.dump
$ sfdisk --force /dev/vda < mbr.dump

The above command updates the CHS addresses in the partition table which seems to trip the bootloader code. sfdisk is somewhat confused about the CHS geometry and 'corrects' it when writing back the partition table. I'm not exactly sure why and how it decides to do this but it seems to cause problems for the installed bootloader.

Before repartitioning:

# sfdisk -l /dev/vda
sfdisk: Disk /dev/vda: cannot get geometry

Disk /dev/vda: 652 cylinders, 255 heads, 63 sectors/track
sfdisk: Warning: The partition table looks like it was made
  for C/H/S=*/147/20 (instead of 652/255/63).
For this listing I'll assume that geometry.
Units: cylinders of 1505280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/vda1   *      0+   2090-   2090-   3072000   83  Linux
		start: (c,h,s) expected (0,102,9) found (0,32,33)
		end: (c,h,s) expected (1023,146,20) found (382,146,20)
/dev/vda2          0       -       0          0    0  Empty
/dev/vda3          0       -       0          0    0  Empty
/dev/vda4          0       -       0          0    0  Empty

I'm not familiar with the bootloader code so can't say if it's a parted bootloader problem or if the extlinux bootloader is simply more forgiving and/or robust. As Lars mentioned, using the extlinux bootloader 'fixes' the issues so I second Matt's suggestion to dd it to the disk during the anaconda post-install stage.

(In reply to Juerg Haefliger from comment #11)
> Growpart is not doing anything wrong AFAICT. In fact, disabling growroot and
> running the following already breaks the subsequent boot:
> $ sfdisk -d /dev/vda > mbr.dump
> $ sfdisk --force /dev/vda < mbr.dump
> 
> The above command updates the CHS addresses in the partition table which
> seems to trip the bootloader code. sfdisk is somewhat confused about the CHS
> geometry and 'corrects' it when writing back the partition table. I'm not
> exactly sure why and how it decides to do this but it seems to cause
> problems for the installed bootloader.


Hmmmm; maybe we should switch growpart to using parted? (Ugh, that kind of looks like a rewrite and not a small bugfix.)

Since we're in the freeze now, I'll put the post-install workaround into place if people agree. Alternately, we could update anaconda to always install the extlinux MBR if extlinux is used.

(In reply to Matthew Miller from comment #12)

> Since we're in the freeze now, I'll put the post-install workaround into
> place if people agree. 

Agreed. We should probably do this for now.


> Alternately, we could update anaconda to always
> install the extlinux MBR if extlinux is used.


I think this gets back to the subject of https://bugzilla.redhat.com/show_bug.cgi?id=1015931

The question is why does this break the gparted bootloader? I tried both sfdisk and fdisk, same result.

So, this is _probably_ really an sfdisk bug. But we've got a lot of different possible workarounds:

1. install extlinux mbr in anaconda (bug #1015931)
2. install extlinux mbr in kickstart post (easiest!)
3. use parted instead of sfdisk in grow-part
4. use parted instead of grow-part directly from cloud-init

I was excited to see that parted 3.2 (now in F21) includes resizepart support directly. And then those hopes for an easy fix there were kind of dashed by upstream cloud-init https://bugs.launchpad.net/cloud-init/+bug/1212492, where parted resizing has been completely removed because it is apparently broken. (Hooray! Everything is broken!)

That probably rules out options 3 and 4. At this stage, my proposal is to implement 2 for F21 and to plan for 1 for F22 and beyond.

Oh yeah, I left "fix sfdisk" off the list. But I guess maybe that's a possibility?

Anyway, I committed the `dd if=/usr/share/syslinux/mbr.bin of=/dev/vda` workaround to the kickstart files for now. This should resolve this bug for now.

Note that the f21 branch of spin-kickstarts was fubar'd before Matt did his commit. After going back before the incorrect merge, Matt's commit no longer cleanly applied to that branch. So that needs to get looked at and committed again.

(In reply to Bruno Wolff III from comment #18)
> Note that the f21 branch of spin-kickstarts was fubar'd before Matt did his
> commit. After going back before the incorrect merge, Matt's commit no longer
> cleanly applied to that branch. So that needs to get looked at and committed
> again.

Thanks Bruno. Done.

MODIFIED per c#17. This change should be in TC4/RC1.

We are still waiting for a TC4 cloud image to be created.

We still need the bug to be in MODIFIED state to indicate that we can compose (the issue is 'addressed' in that we believe when an image actually gets composed, the bug will be fixed).

We're still blocked in actual testing by bug #1156603 (compose failure) but I've tested the kickstart with the fix with anaconda _locally_ and it seems to work fine, including rebooting.

I tested the RC1 builds Dennis just made and the fix works. Presumably will also work in RC2. :)

This has been confirmed fixed in RC1 and RC4 by multiple testers.

For the record, we have been looking into this area again recently. We have also confirmed that this bug still affects Fedora 22 if the 'dd' line is removed from cloud kickstart %post.

We basically reconstructed and reconfirmed Lars' work from #c6, with an additional wrinkle. Dusty found that simply doing this:

sfdisk --dump /dev/vda > dump
xxd -l512 /dev/vda > before
sfdisk /dev/vda < dump --force
xxd -l512 /dev/vda > after

is enough to cause a change to the disk label - 'before' and 'after' are not identical. That seems clearly a bug in sfdisk, as this command should only dump and reload the existing configuration. It seems this is the change that prevents the parted-deployed boot sector code from working. He also noted this difference in the output of 'file /dev/vda' before and after the dump/restore:

------------

# Before sfdisk dump/restore sfdisk -d /dev/vda > mbr.dump; sfdisk --force /dev/vda < mbr.dump;
[root@f22sfdisk ~]# file -s /dev/vda
/dev/vda: DOS/MBR boot sector; partition 1 : ID=0x83, active, start-CHS (0x0,32,33), end-CHS (0x17e,146,20), startsector 2048, 6144000 sectors

# After sfdisk dump/restore
[root@f22sfdisk ~]# file -s /dev/vda
/dev/vda: DOS/MBR boot sector; partition 1 : ID=0x83, active, start-CHS (0x2,0,33), end-CHS (0x3d1,4,20), startsector 2048, 6144000 sectors

------------

I'm going to file a util-linux bug.

util-linux bug: https://bugzilla.redhat.com/show_bug.cgi?id=1211405