Description of problem: cloud-init[768]: 2016-04-14 19:52:11,207 - util.py[WARNING]: Failed: growpart /dev/vda 1 and the root filesystem is not resized Version-Release number of selected component (if applicable): Fedora-Cloud-Base-Rawhide-20160414 How reproducible: Every time Steps to Reproduce: 1. Boot fedora rawhide qcow2 image 2. cloud-init shows growpart failed 3. Actual results: df -h shows root partition is not resized on first boot Expected results: df -h shows root partition is resized on first boot Additional info: # df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 2.0G 0 2.0G 0% /dev tmpfs 2.0G 0 2.0G 0% /dev/shm tmpfs 2.0G 396K 2.0G 1% /run tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup /dev/vda1 2.9G 529M 2.3G 19% / tmpfs 396M 0 396M 0% /run/user/1000 # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 252:0 0 10G 0 disk └─vda1 252:1 0 3G 0 part / Partition is apparently not resized, however # fdisk /dev/vda Welcome to fdisk (util-linux 2.28). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Command (m for help): p Disk /dev/vda: 10 GiB, 10737418240 bytes, 20971520 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x8672c2e4 Device Boot Start End Sectors Size Id Type /dev/vda1 * 2048 20971519 20969472 10G 83 Linux ===== Reading straight from the partition table, is has been resized. After a reboot, the root partition is resized since the partition table has been adjusted.
I just ran into the same issue with a new cloud image that had the latest util-linux package (2.28-1.fc23) reverting to util-linux 2.27 fixed the problem.
Seeing this on Fedora 23 w/ cloud-utils-growpart-0.27-14.fc23.noarch and util-linux-2.28-1.fc23.x86_64. Confirmed downgrading to util-linux-2.27 fixes it. Here's what my journal says when this happens: Apr 26 13:17:03 localhost.localdomain cloud-init[652]: [CLOUDINIT] util.py[WARNING]: Failed: growpart /dev/vda 1 Apr 26 13:17:03 localhost.localdomain cloud-init[652]: [CLOUDINIT] util.py[DEBUG]: Failed: growpart /dev/vda 1 Traceback (most recent call last): File "/usr/lib/python3.4/site-packages/cloudinit/config/cc_growpart.py", line 109, in resize util.subp(["growpart", diskdev, partnum]) File "/usr/lib/python3.4/site-packages/cloudinit/util.py", line 1654, in subp cmd=args) cloudinit.util.ProcessExecutionError: Unexpected error while running command. Command: ['growpart', '/dev/vda', '1'] Exit code: 2 Reason: - Stdout: 'FAILED: pt_resize failed\n' Stderr: 'failed [pt_update:1] pt_update /dev/vda 1\npartx: partition and disk name do not match\n' Looks like this error message was added 3/22: https://github.com/karelzak/util-linux/commit/2d47fa3921b86f16b07065a83e381708fa852d4a#diff-9352df20ce8f48680ecee861bbcccbf8R923
Ran into this one too. Here's some extra info that may help.. From the journal: May 10 15:18:43 shtest-20160510-2216 cloud-init[1068]: [CLOUDINIT] util.py[DEBUG]: Running command ['growpart', '/dev/xvda', '1'] with allowed return codes [0] (shell=False, capture=True) From an strace of "growpart /dev/xvda 1": 21016 execve("/usr/sbin/partx", ["partx", "--update", "1", "/dev/xvda"], [/* 24 vars */]) = 0 [root@shtest-20160510-2216 ~]# rpm -q util-linux libblkid libuuid libfdisk libmount libsmartcols util-linux-2.27-1.fc23.x86_64 libblkid-2.27-1.fc23.x86_64 libuuid-2.27-1.fc23.x86_64 libfdisk-2.27-1.fc23.x86_64 libmount-2.27-1.fc23.x86_64 libsmartcols-2.27-1.fc23.x86_64 [root@shtest-20160510-2110 ~]# partx --update 1 /dev/xvda [root@shtest-20160510-2110 ~]# echo $? 0 [root@shtest-20160510-2216 ~]# dnf -y update util-linux libblkid libuuid libfdisk libmount libsmartcols Last metadata expiration check: 1:13:20 ago on Tue May 10 14:34:46 2016. Dependencies resolved. ==================================================================================================== Package Arch Version Repository Size ==================================================================================================== Upgrading: libblkid x86_64 2.28-1.fc23 updates 179 k libfdisk x86_64 2.28-1.fc23 updates 221 k libmount x86_64 2.28-1.fc23 updates 198 k libsmartcols x86_64 2.28-1.fc23 updates 142 k libuuid x86_64 2.28-1.fc23 updates 79 k util-linux x86_64 2.28-1.fc23 updates 2.2 M Transaction Summary ==================================================================================================== Upgrade 6 Packages Total download size: 3.0 M Downloading Packages: (1/6): libblkid-2.28-1.fc23.x86_64.rpm 3.4 MB/s | 179 kB 00:00 (2/6): libuuid-2.28-1.fc23.x86_64.rpm 1.2 MB/s | 79 kB 00:00 (3/6): libmount-2.28-1.fc23.x86_64.rpm 2.8 MB/s | 198 kB 00:00 (4/6): libfdisk-2.28-1.fc23.x86_64.rpm 4.2 MB/s | 221 kB 00:00 (5/6): libsmartcols-2.28-1.fc23.x86_64.rpm 2.2 MB/s | 142 kB 00:00 (6/6): util-linux-2.28-1.fc23.x86_64.rpm 14 MB/s | 2.2 MB 00:00 ---------------------------------------------------------------------------------------------------- Total 14 MB/s | 3.0 MB 00:00 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Upgrading : libuuid-2.28-1.fc23.x86_64 1/12 Upgrading : libblkid-2.28-1.fc23.x86_64 2/12 Upgrading : libmount-2.28-1.fc23.x86_64 3/12 Upgrading : libfdisk-2.28-1.fc23.x86_64 4/12 Upgrading : libsmartcols-2.28-1.fc23.x86_64 5/12 Upgrading : util-linux-2.28-1.fc23.x86_64 6/12 Cleanup : util-linux-2.27-1.fc23.x86_64 7/12 Cleanup : libfdisk-2.27-1.fc23.x86_64 8/12 Cleanup : libmount-2.27-1.fc23.x86_64 9/12 Cleanup : libblkid-2.27-1.fc23.x86_64 10/12 Cleanup : libuuid-2.27-1.fc23.x86_64 11/12 Cleanup : libsmartcols-2.27-1.fc23.x86_64 12/12 Verifying : libblkid-2.28-1.fc23.x86_64 1/12 Verifying : libuuid-2.28-1.fc23.x86_64 2/12 Verifying : libmount-2.28-1.fc23.x86_64 3/12 Verifying : util-linux-2.28-1.fc23.x86_64 4/12 Verifying : libfdisk-2.28-1.fc23.x86_64 5/12 Verifying : libsmartcols-2.28-1.fc23.x86_64 6/12 Verifying : libmount-2.27-1.fc23.x86_64 7/12 Verifying : libfdisk-2.27-1.fc23.x86_64 8/12 Verifying : libuuid-2.27-1.fc23.x86_64 9/12 Verifying : libblkid-2.27-1.fc23.x86_64 10/12 Verifying : util-linux-2.27-1.fc23.x86_64 11/12 Verifying : libsmartcols-2.27-1.fc23.x86_64 12/12 Upgraded: libblkid.x86_64 2.28-1.fc23 libfdisk.x86_64 2.28-1.fc23 libmount.x86_64 2.28-1.fc23 libsmartcols.x86_64 2.28-1.fc23 libuuid.x86_64 2.28-1.fc23 util-linux.x86_64 2.28-1.fc23 Complete! [root@shtest-20160510-2216 ~]# rpm -q util-linux libblkid libuuid libfdisk libmount libsmartcolsutil-linux-2.28-1.fc23.x86_64 libblkid-2.28-1.fc23.x86_64 libuuid-2.28-1.fc23.x86_64 libfdisk-2.28-1.fc23.x86_64 libmount-2.28-1.fc23.x86_64 libsmartcols-2.28-1.fc23.x86_64 [root@shtest-20160510-2216 ~]# partx --update 1 /dev/xvda partx: partition and disk name do not match [root@shtest-20160510-2216 ~]# echo $? 1
Fedora-Cloud-Base-24_Beta-1.6.x86_64 has the same resize issue. Likely the system failed(or did not tried) to reread the partition table after editing. If I execute the $ sudo partprobe It is working, and I am able to manually resize the filesystem vie resize2fs. I hope someone will have time to find the root cause before the release, this issue looks critical.
Proposed as a Blocker for 24-final by Fedora user afazekas using the blocker tracking app because: The automatic resize of the root filesystem at the first boot is critically important feature of a cloud-image.
The criterion here is: "Release blocking cloud images must be able to automatically utilize all available space on a supported volume" - https://fedoraproject.org/wiki/Fedora_24_Beta_Release_Criteria#Growroot +1 blocker, cloud folks, can you look into this? Thanks.
On a process note...we don't appear to have a test case for this in the Cloud matrix - https://fedoraproject.org/wiki/Template:Cloud_test_matrix - unless one of those test cases covers it and I didn't notice. Kushal, does autocloud test this?
This looks like a problem with how growpart is calling partx. Partx wants: partx --update /dev/vda1 Or: partx --update --nr 1 /dev/vda But growpart is calling: partx --update /dev/vda 1 Which yields: partx: partition and disk name do not match It looks like this may stem from a recent change in util-linux; see for example http://www.spinics.net/lists/util-linux-ng/msg12668.html. My reading of that message is that the syntax used by growpart was always incorrect, but worked because there was insufficient error checking in partx. This should be submitted upstream (https://launchpad.net/cloud-utils/).
Never mind, I totally lied with that last comment; I was misreading the word in front of my face. I can't actually reproduce this.
I've reported this upstream as https://bugs.launchpad.net/cloud-utils/+bug/1587971, and I've submitted a patch. My description of how partx was being called in #8 was incorrect, but the error message (obviously) and solution remain the same. For bonus points, the syntax that works with 2.28 (partx --update --nr 1 /dev/vda) also works with < 2.28.
growpart is part of cloud-utils. Reassigning...
cloud-utils has not been touched since 2015 and we're close to F24 go/no-go, so I'm gonna use my provenpackager privs to fix this to move the process along.
+1 blocker
cloud-utils-0.27-16.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-34ec440575
cloud-utils-0.27-16.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-714119426f
I sent out an update for F23 as well, as it looks to me like the newer util-linux has been sent out as an F23 update and so newly-built F23 cloud images are probably broken too.
+1 blocker per Release Criterion cited on comment 6
that's four +1s, setting AcceptedBlocker.
cloud-utils-0.27-16.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-714119426f
It worked for a latest nightly rebuilt of Fedora 24 Cloud Base image on my local test system. Thank you for updating this. On a side note: we don't have a test for this yet. We will add this in future (as we did for other test cases).
I would be willing to test if there is a cloud base compose with this new package. I tried the 0607 fc24 build and it is still -15. Do the nightly composes pull packages from testing? Since the problem only manifests on first boot, the package needs to be in the image from the start.
"Do the nightly composes pull packages from testing?" No, which leads to this kind of catch-22 problem with any package that you need an image to test (we have the same issue with anaconda). We have some thoughts about solving that, but nothing implemented yet, the only options are to build your own images to test, or come up with some kind of equivalent testing scenario if you can.
This was reported verified in Bodhi.
cloud-utils-0.27-16.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.