Description of problem:
cloud-init: 2016-04-14 19:52:11,207 - util.py[WARNING]: Failed: growpart /dev/vda 1
and the root filesystem is not resized
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Boot fedora rawhide qcow2 image
2. cloud-init shows growpart failed
df -h shows root partition is not resized on first boot
df -h shows root partition is resized on first boot
# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 2.0G 0 2.0G 0% /dev
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 2.0G 396K 2.0G 1% /run
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/vda1 2.9G 529M 2.3G 19% /
tmpfs 396M 0 396M 0% /run/user/1000
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 252:0 0 10G 0 disk
└─vda1 252:1 0 3G 0 part /
Partition is apparently not resized, however
# fdisk /dev/vda
Welcome to fdisk (util-linux 2.28).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Command (m for help): p
Disk /dev/vda: 10 GiB, 10737418240 bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x8672c2e4
Device Boot Start End Sectors Size Id Type
/dev/vda1 * 2048 20971519 20969472 10G 83 Linux
Reading straight from the partition table, is has been resized.
After a reboot, the root partition is resized since the partition table has been adjusted.
I just ran into the same issue with a new cloud image that had the latest util-linux package (2.28-1.fc23) reverting to util-linux 2.27 fixed the problem.
Seeing this on Fedora 23 w/ cloud-utils-growpart-0.27-14.fc23.noarch and util-linux-2.28-1.fc23.x86_64. Confirmed downgrading to util-linux-2.27 fixes it.
Here's what my journal says when this happens:
Apr 26 13:17:03 localhost.localdomain cloud-init: [CLOUDINIT] util.py[WARNING]: Failed: growpart /dev/vda 1
Apr 26 13:17:03 localhost.localdomain cloud-init: [CLOUDINIT] util.py[DEBUG]: Failed: growpart /dev/vda 1
Traceback (most recent call last):
File "/usr/lib/python3.4/site-packages/cloudinit/config/cc_growpart.py", line 109, in resize
util.subp(["growpart", diskdev, partnum])
File "/usr/lib/python3.4/site-packages/cloudinit/util.py", line 1654, in subp
cloudinit.util.ProcessExecutionError: Unexpected error while running command.
Command: ['growpart', '/dev/vda', '1']
Exit code: 2
Stdout: 'FAILED: pt_resize failed\n'
Stderr: 'failed [pt_update:1] pt_update /dev/vda 1\npartx: partition and disk name do not match\n'
Looks like this error message was added 3/22:
Ran into this one too. Here's some extra info that may help..
From the journal:
May 10 15:18:43 shtest-20160510-2216 cloud-init: [CLOUDINIT] util.py[DEBUG]: Running command ['growpart', '/dev/xvda', '1'] with allowed return codes  (shell=False, capture=True)
From an strace of "growpart /dev/xvda 1":
21016 execve("/usr/sbin/partx", ["partx", "--update", "1", "/dev/xvda"], [/* 24 vars */]) = 0
[root@shtest-20160510-2216 ~]# rpm -q util-linux libblkid libuuid libfdisk libmount libsmartcols
[root@shtest-20160510-2110 ~]# partx --update 1 /dev/xvda
[root@shtest-20160510-2110 ~]# echo $?
[root@shtest-20160510-2216 ~]# dnf -y update util-linux libblkid libuuid libfdisk libmount libsmartcols
Last metadata expiration check: 1:13:20 ago on Tue May 10 14:34:46 2016.
Package Arch Version Repository Size
libblkid x86_64 2.28-1.fc23 updates 179 k
libfdisk x86_64 2.28-1.fc23 updates 221 k
libmount x86_64 2.28-1.fc23 updates 198 k
libsmartcols x86_64 2.28-1.fc23 updates 142 k
libuuid x86_64 2.28-1.fc23 updates 79 k
util-linux x86_64 2.28-1.fc23 updates 2.2 M
Upgrade 6 Packages
Total download size: 3.0 M
(1/6): libblkid-2.28-1.fc23.x86_64.rpm 3.4 MB/s | 179 kB 00:00
(2/6): libuuid-2.28-1.fc23.x86_64.rpm 1.2 MB/s | 79 kB 00:00
(3/6): libmount-2.28-1.fc23.x86_64.rpm 2.8 MB/s | 198 kB 00:00
(4/6): libfdisk-2.28-1.fc23.x86_64.rpm 4.2 MB/s | 221 kB 00:00
(5/6): libsmartcols-2.28-1.fc23.x86_64.rpm 2.2 MB/s | 142 kB 00:00
(6/6): util-linux-2.28-1.fc23.x86_64.rpm 14 MB/s | 2.2 MB 00:00
Total 14 MB/s | 3.0 MB 00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Upgrading : libuuid-2.28-1.fc23.x86_64 1/12
Upgrading : libblkid-2.28-1.fc23.x86_64 2/12
Upgrading : libmount-2.28-1.fc23.x86_64 3/12
Upgrading : libfdisk-2.28-1.fc23.x86_64 4/12
Upgrading : libsmartcols-2.28-1.fc23.x86_64 5/12
Upgrading : util-linux-2.28-1.fc23.x86_64 6/12
Cleanup : util-linux-2.27-1.fc23.x86_64 7/12
Cleanup : libfdisk-2.27-1.fc23.x86_64 8/12
Cleanup : libmount-2.27-1.fc23.x86_64 9/12
Cleanup : libblkid-2.27-1.fc23.x86_64 10/12
Cleanup : libuuid-2.27-1.fc23.x86_64 11/12
Cleanup : libsmartcols-2.27-1.fc23.x86_64 12/12
Verifying : libblkid-2.28-1.fc23.x86_64 1/12
Verifying : libuuid-2.28-1.fc23.x86_64 2/12
Verifying : libmount-2.28-1.fc23.x86_64 3/12
Verifying : util-linux-2.28-1.fc23.x86_64 4/12
Verifying : libfdisk-2.28-1.fc23.x86_64 5/12
Verifying : libsmartcols-2.28-1.fc23.x86_64 6/12
Verifying : libmount-2.27-1.fc23.x86_64 7/12
Verifying : libfdisk-2.27-1.fc23.x86_64 8/12
Verifying : libuuid-2.27-1.fc23.x86_64 9/12
Verifying : libblkid-2.27-1.fc23.x86_64 10/12
Verifying : util-linux-2.27-1.fc23.x86_64 11/12
Verifying : libsmartcols-2.27-1.fc23.x86_64 12/12
libblkid.x86_64 2.28-1.fc23 libfdisk.x86_64 2.28-1.fc23 libmount.x86_64 2.28-1.fc23
libsmartcols.x86_64 2.28-1.fc23 libuuid.x86_64 2.28-1.fc23 util-linux.x86_64 2.28-1.fc23
[root@shtest-20160510-2216 ~]# rpm -q util-linux libblkid libuuid libfdisk libmount libsmartcolsutil-linux-2.28-1.fc23.x86_64
[root@shtest-20160510-2216 ~]# partx --update 1 /dev/xvda
partx: partition and disk name do not match
[root@shtest-20160510-2216 ~]# echo $?
Fedora-Cloud-Base-24_Beta-1.6.x86_64 has the same resize issue.
Likely the system failed(or did not tried) to reread the partition table after editing.
If I execute the
$ sudo partprobe
It is working, and I am able to manually resize the filesystem vie resize2fs.
I hope someone will have time to find the root cause before the release,
this issue looks critical.
Proposed as a Blocker for 24-final by Fedora user afazekas using the blocker tracking app because:
The automatic resize of the root filesystem at the first boot is critically important feature of a cloud-image.
The criterion here is: "Release blocking cloud images must be able to automatically utilize all available space on a supported volume" - https://fedoraproject.org/wiki/Fedora_24_Beta_Release_Criteria#Growroot
+1 blocker, cloud folks, can you look into this? Thanks.
On a process note...we don't appear to have a test case for this in the Cloud matrix - https://fedoraproject.org/wiki/Template:Cloud_test_matrix - unless one of those test cases covers it and I didn't notice. Kushal, does autocloud test this?
This looks like a problem with how growpart is calling partx. Partx wants:
partx --update /dev/vda1
partx --update --nr 1 /dev/vda
But growpart is calling:
partx --update /dev/vda 1
partx: partition and disk name do not match
It looks like this may stem from a recent change in util-linux; see for example http://www.spinics.net/lists/util-linux-ng/msg12668.html. My reading of that message is that the syntax used by growpart was always incorrect, but worked because there was insufficient error checking in partx.
This should be submitted upstream (https://launchpad.net/cloud-utils/).
Never mind, I totally lied with that last comment; I was misreading the word in front of my face. I can't actually reproduce this.
I've reported this upstream as https://bugs.launchpad.net/cloud-utils/+bug/1587971, and I've submitted a patch. My description of how partx was being called in #8 was incorrect, but the error message (obviously) and solution remain the same.
For bonus points, the syntax that works with 2.28 (partx --update --nr 1 /dev/vda) also works with < 2.28.
growpart is part of cloud-utils. Reassigning...
cloud-utils has not been touched since 2015 and we're close to F24 go/no-go, so I'm gonna use my provenpackager privs to fix this to move the process along.
cloud-utils-0.27-16.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-34ec440575
cloud-utils-0.27-16.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-714119426f
I sent out an update for F23 as well, as it looks to me like the newer util-linux has been sent out as an F23 update and so newly-built F23 cloud images are probably broken too.
+1 blocker per Release Criterion cited on comment 6
that's four +1s, setting AcceptedBlocker.
cloud-utils-0.27-16.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-714119426f
It worked for a latest nightly rebuilt of Fedora 24 Cloud Base image on my local test system. Thank you for updating this.
On a side note: we don't have a test for this yet. We will add this in future (as we did for other test cases).
I would be willing to test if there is a cloud base compose with this new package. I tried the 0607 fc24 build and it is still -15. Do the nightly composes pull packages from testing?
Since the problem only manifests on first boot, the package needs to be in the image from the start.
"Do the nightly composes pull packages from testing?"
No, which leads to this kind of catch-22 problem with any package that you need an image to test (we have the same issue with anaconda). We have some thoughts about solving that, but nothing implemented yet, the only options are to build your own images to test, or come up with some kind of equivalent testing scenario if you can.
This was reported verified in Bodhi.
cloud-utils-0.27-16.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.