Bug 1327337 - util.py[WARNING]: Failed: growpart /dev/vda 1
Summary: util.py[WARNING]: Failed: growpart /dev/vda 1
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: cloud-utils
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Juerg Haefliger
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedBlocker
Depends On:
Blocks: F24FinalBlocker
TreeView+ depends on / blocked
 
Reported: 2016-04-14 20:22 UTC by Seth Jennings
Modified: 2016-06-09 14:09 UTC (History)
20 users (show)

Fixed In Version: cloud-utils-0.27-16.fc24
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-06-09 14:09:43 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Launchpad 1587971 None None None 2016-06-01 16:28:01 UTC

Description Seth Jennings 2016-04-14 20:22:27 UTC
Description of problem:
cloud-init[768]: 2016-04-14 19:52:11,207 - util.py[WARNING]: Failed: growpart /dev/vda 1
and the root filesystem is not resized

Version-Release number of selected component (if applicable):
Fedora-Cloud-Base-Rawhide-20160414

How reproducible:
Every time

Steps to Reproduce:
1. Boot fedora rawhide qcow2 image
2. cloud-init shows growpart failed
3.

Actual results:
df -h shows root partition is not resized on first boot

Expected results:
df -h shows root partition is resized on first boot

Additional info:

# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        2.0G     0  2.0G   0% /dev
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           2.0G  396K  2.0G   1% /run
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/vda1       2.9G  529M  2.3G  19% /
tmpfs           396M     0  396M   0% /run/user/1000

# lsblk
NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda    252:0    0  10G  0 disk 
└─vda1 252:1    0   3G  0 part /

Partition is apparently not resized, however

# fdisk /dev/vda

Welcome to fdisk (util-linux 2.28).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): p
Disk /dev/vda: 10 GiB, 10737418240 bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x8672c2e4

Device     Boot Start      End  Sectors Size Id Type
/dev/vda1  *     2048 20971519 20969472  10G 83 Linux
=====

Reading straight from the partition table, is has been resized.

After a reboot, the root partition is resized since the partition table has been adjusted.

Comment 1 Oskari Saarenmaa 2016-04-21 16:13:09 UTC
I just ran into the same issue with a new cloud image that had the latest util-linux package (2.28-1.fc23) reverting to util-linux 2.27 fixed the problem.

Comment 2 Dan Peschman 2016-04-27 01:54:36 UTC
Seeing this on Fedora 23 w/ cloud-utils-growpart-0.27-14.fc23.noarch and util-linux-2.28-1.fc23.x86_64.  Confirmed downgrading to util-linux-2.27 fixes it.

Here's what my journal says when this happens:

Apr 26 13:17:03 localhost.localdomain cloud-init[652]: [CLOUDINIT] util.py[WARNING]: Failed: growpart /dev/vda 1
Apr 26 13:17:03 localhost.localdomain cloud-init[652]: [CLOUDINIT] util.py[DEBUG]: Failed: growpart /dev/vda 1
Traceback (most recent call last):
 File "/usr/lib/python3.4/site-packages/cloudinit/config/cc_growpart.py", line 109, in resize
   util.subp(["growpart", diskdev, partnum])
 File "/usr/lib/python3.4/site-packages/cloudinit/util.py", line 1654, in subp
   cmd=args)
cloudinit.util.ProcessExecutionError: Unexpected error while running command.
Command: ['growpart', '/dev/vda', '1']
Exit code: 2
Reason: -
Stdout: 'FAILED: pt_resize failed\n'
Stderr: 'failed [pt_update:1] pt_update /dev/vda 1\npartx: partition and disk name do not match\n'

Looks like this error message was added 3/22:
https://github.com/karelzak/util-linux/commit/2d47fa3921b86f16b07065a83e381708fa852d4a#diff-9352df20ce8f48680ecee861bbcccbf8R923

Comment 3 Shayne Hardesty 2016-05-10 16:04:20 UTC
Ran into this one too.  Here's some extra info that may help..

From the journal:
May 10 15:18:43 shtest-20160510-2216 cloud-init[1068]: [CLOUDINIT] util.py[DEBUG]: Running command ['growpart', '/dev/xvda', '1'] with allowed return codes [0] (shell=False, capture=True)

From an strace of "growpart /dev/xvda 1":
21016 execve("/usr/sbin/partx", ["partx", "--update", "1", "/dev/xvda"], [/* 24 vars */]) = 0

[root@shtest-20160510-2216 ~]# rpm -q util-linux libblkid libuuid libfdisk libmount libsmartcols
util-linux-2.27-1.fc23.x86_64
libblkid-2.27-1.fc23.x86_64
libuuid-2.27-1.fc23.x86_64
libfdisk-2.27-1.fc23.x86_64
libmount-2.27-1.fc23.x86_64
libsmartcols-2.27-1.fc23.x86_64

[root@shtest-20160510-2110 ~]# partx --update 1 /dev/xvda
[root@shtest-20160510-2110 ~]# echo $?
0

[root@shtest-20160510-2216 ~]# dnf -y update util-linux libblkid libuuid libfdisk libmount libsmartcols
Last metadata expiration check: 1:13:20 ago on Tue May 10 14:34:46 2016.
Dependencies resolved.
====================================================================================================
 Package                   Arch                Version                   Repository            Size
====================================================================================================
Upgrading:
 libblkid                  x86_64              2.28-1.fc23               updates              179 k
 libfdisk                  x86_64              2.28-1.fc23               updates              221 k
 libmount                  x86_64              2.28-1.fc23               updates              198 k
 libsmartcols              x86_64              2.28-1.fc23               updates              142 k
 libuuid                   x86_64              2.28-1.fc23               updates               79 k
 util-linux                x86_64              2.28-1.fc23               updates              2.2 M

Transaction Summary
====================================================================================================
Upgrade  6 Packages

Total download size: 3.0 M
Downloading Packages:
(1/6): libblkid-2.28-1.fc23.x86_64.rpm                              3.4 MB/s | 179 kB     00:00
(2/6): libuuid-2.28-1.fc23.x86_64.rpm                               1.2 MB/s |  79 kB     00:00
(3/6): libmount-2.28-1.fc23.x86_64.rpm                              2.8 MB/s | 198 kB     00:00
(4/6): libfdisk-2.28-1.fc23.x86_64.rpm                              4.2 MB/s | 221 kB     00:00
(5/6): libsmartcols-2.28-1.fc23.x86_64.rpm                          2.2 MB/s | 142 kB     00:00
(6/6): util-linux-2.28-1.fc23.x86_64.rpm                             14 MB/s | 2.2 MB     00:00
----------------------------------------------------------------------------------------------------
Total                                                                14 MB/s | 3.0 MB     00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Upgrading   : libuuid-2.28-1.fc23.x86_64                                                     1/12
  Upgrading   : libblkid-2.28-1.fc23.x86_64                                                    2/12
  Upgrading   : libmount-2.28-1.fc23.x86_64                                                    3/12
  Upgrading   : libfdisk-2.28-1.fc23.x86_64                                                    4/12
  Upgrading   : libsmartcols-2.28-1.fc23.x86_64                                                5/12
  Upgrading   : util-linux-2.28-1.fc23.x86_64                                                  6/12
  Cleanup     : util-linux-2.27-1.fc23.x86_64                                                  7/12
  Cleanup     : libfdisk-2.27-1.fc23.x86_64                                                    8/12
  Cleanup     : libmount-2.27-1.fc23.x86_64                                                    9/12
  Cleanup     : libblkid-2.27-1.fc23.x86_64                                                   10/12
  Cleanup     : libuuid-2.27-1.fc23.x86_64                                                    11/12
  Cleanup     : libsmartcols-2.27-1.fc23.x86_64                                               12/12
  Verifying   : libblkid-2.28-1.fc23.x86_64                                                    1/12
  Verifying   : libuuid-2.28-1.fc23.x86_64                                                     2/12
  Verifying   : libmount-2.28-1.fc23.x86_64                                                    3/12
  Verifying   : util-linux-2.28-1.fc23.x86_64                                                  4/12
  Verifying   : libfdisk-2.28-1.fc23.x86_64                                                    5/12
  Verifying   : libsmartcols-2.28-1.fc23.x86_64                                                6/12
  Verifying   : libmount-2.27-1.fc23.x86_64                                                    7/12
  Verifying   : libfdisk-2.27-1.fc23.x86_64                                                    8/12
  Verifying   : libuuid-2.27-1.fc23.x86_64                                                     9/12
  Verifying   : libblkid-2.27-1.fc23.x86_64                                                   10/12
  Verifying   : util-linux-2.27-1.fc23.x86_64                                                 11/12
  Verifying   : libsmartcols-2.27-1.fc23.x86_64                                               12/12

Upgraded:
  libblkid.x86_64 2.28-1.fc23        libfdisk.x86_64 2.28-1.fc23    libmount.x86_64 2.28-1.fc23
  libsmartcols.x86_64 2.28-1.fc23    libuuid.x86_64 2.28-1.fc23     util-linux.x86_64 2.28-1.fc23

Complete!

[root@shtest-20160510-2216 ~]# rpm -q util-linux libblkid libuuid libfdisk libmount libsmartcolsutil-linux-2.28-1.fc23.x86_64
libblkid-2.28-1.fc23.x86_64
libuuid-2.28-1.fc23.x86_64
libfdisk-2.28-1.fc23.x86_64
libmount-2.28-1.fc23.x86_64
libsmartcols-2.28-1.fc23.x86_64

[root@shtest-20160510-2216 ~]# partx --update 1 /dev/xvda
partx: partition and disk name do not match
[root@shtest-20160510-2216 ~]# echo $?
1

Comment 4 Attila Fazekas 2016-06-01 14:19:43 UTC
Fedora-Cloud-Base-24_Beta-1.6.x86_64 has the same resize issue.

Likely the system failed(or did not tried) to reread the partition table after editing.

If I execute the
$ sudo partprobe

It is working, and I am able to manually resize the filesystem vie resize2fs.

I hope someone will have time to find the root cause before the release,
this issue looks critical.

Comment 5 Fedora Blocker Bugs Application 2016-06-01 15:40:06 UTC
Proposed as a Blocker for 24-final by Fedora user afazekas using the blocker tracking app because:

 The automatic resize of the root filesystem at the first boot is critically important feature of a cloud-image.

Comment 6 Adam Williamson 2016-06-01 15:48:39 UTC
The criterion here is: "Release blocking cloud images must be able to automatically utilize all available space on a supported volume" - https://fedoraproject.org/wiki/Fedora_24_Beta_Release_Criteria#Growroot

+1 blocker, cloud folks, can you look into this? Thanks.

Comment 7 Adam Williamson 2016-06-01 15:50:33 UTC
On a process note...we don't appear to have a test case for this in the Cloud matrix - https://fedoraproject.org/wiki/Template:Cloud_test_matrix - unless one of those test cases covers it and I didn't notice. Kushal, does autocloud test this?

Comment 8 Lars Kellogg-Stedman 2016-06-01 15:57:58 UTC
This looks like a problem with how growpart is calling partx.  Partx wants:

    partx --update /dev/vda1

Or:

    partx --update --nr 1 /dev/vda

But growpart is calling:

    partx --update /dev/vda 1

Which yields:

    partx: partition and disk name do not match

It looks like this may stem from a recent change in util-linux; see for example http://www.spinics.net/lists/util-linux-ng/msg12668.html.  My reading of that message is that the syntax used by growpart was always incorrect, but worked because there was insufficient error checking in partx.

This should be submitted upstream (https://launchpad.net/cloud-utils/).

Comment 9 Lars Kellogg-Stedman 2016-06-01 16:13:03 UTC
Never mind, I totally lied with that last comment; I was misreading the word in front of my face.  I can't actually reproduce this.

Comment 10 Lars Kellogg-Stedman 2016-06-01 16:28:02 UTC
I've reported this upstream as https://bugs.launchpad.net/cloud-utils/+bug/1587971, and I've submitted a patch.  My description of how partx was being called in #8 was incorrect, but the error message (obviously) and solution remain the same.

For bonus points, the syntax that works with 2.28 (partx --update --nr 1 /dev/vda) also works with < 2.28.

Comment 11 Garrett Holmstrom 2016-06-01 18:17:29 UTC
growpart is part of cloud-utils.  Reassigning...

Comment 12 Adam Williamson 2016-06-03 19:17:56 UTC
cloud-utils has not been touched since 2015 and we're close to F24 go/no-go, so I'm gonna use my provenpackager privs to fix this to move the process along.

Comment 13 Stephen Gallagher 2016-06-03 19:56:14 UTC
+1 blocker

Comment 14 Fedora Update System 2016-06-03 20:39:27 UTC
cloud-utils-0.27-16.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-34ec440575

Comment 15 Fedora Update System 2016-06-03 20:39:41 UTC
cloud-utils-0.27-16.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-714119426f

Comment 16 Adam Williamson 2016-06-03 20:40:45 UTC
I sent out an update for F23 as well, as it looks to me like the newer util-linux has been sent out as an F23 update and so newly-built F23 cloud images are probably broken too.

Comment 17 Giulio 'juliuxpigface' 2016-06-03 21:39:33 UTC
+1 blocker per Release Criterion cited on comment 6

Comment 18 Kevin Fenzi 2016-06-03 21:42:40 UTC
+1 blocker

Comment 19 Adam Williamson 2016-06-03 22:36:01 UTC
that's four +1s, setting AcceptedBlocker.

Comment 20 Fedora Update System 2016-06-04 18:25:24 UTC
cloud-utils-0.27-16.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-714119426f

Comment 21 kushaldas@gmail.com 2016-06-07 20:50:51 UTC
It worked for a latest nightly rebuilt of Fedora 24 Cloud Base image on my local test system. Thank you for updating this.

On a side note: we don't have a test for this yet. We will add this in future (as we did for other test cases).

Comment 22 Seth Jennings 2016-06-07 21:25:37 UTC
I would be willing to test if there is a cloud base compose with this new package.  I tried the 0607 fc24 build and it is still -15.  Do the nightly composes pull packages from testing?

Since the problem only manifests on first boot, the package needs to be in the image from the start.

Comment 23 Adam Williamson 2016-06-07 22:14:40 UTC
"Do the nightly composes pull packages from testing?"

No, which leads to this kind of catch-22 problem with any package that you need an image to test (we have the same issue with anaconda). We have some thoughts about solving that, but nothing implemented yet, the only options are to build your own images to test, or come up with some kind of equivalent testing scenario if you can.

Comment 24 Adam Williamson 2016-06-08 22:07:10 UTC
This was reported verified in Bodhi.

Comment 25 Fedora Update System 2016-06-09 14:09:37 UTC
cloud-utils-0.27-16.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.