Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be unavailable on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1490505 - cloud-init fails when calling `xfs_growfs /dev/mapper/atomicos-root` on Fedora Atomic Host
Summary: cloud-init fails when calling `xfs_growfs /dev/mapper/atomicos-root` on Fedor...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: cloud-init
Version: 27
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Garrett Holmstrom
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedFreezeException
Depends On:
Blocks: F27BetaFreezeException
TreeView+ depends on / blocked
 
Reported: 2017-09-11 19:36 UTC by Dusty Mabe
Modified: 2017-09-25 18:05 UTC (History)
9 users (show)

Fixed In Version: cloud-init-0.7.9-9.fc27
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-09-25 18:05:49 UTC
Type: Bug


Attachments (Terms of Use)

Description Dusty Mabe 2017-09-11 19:36:25 UTC
Description of problem:

cloud-init fails when calling `xfs_growfs /dev/mapper/atomicos-root`:

```
Sep 11 18:15:50 localhost.localdomain cloud-init[729]: 2017-09-11 18:15:50,299 - util.py[WARNING]: Failed to resize filesystem (cmd=('xfs_growfs', '/dev/mapper/atomicos-root'))
Sep 11 18:15:50 localhost.localdomain cloud-init[729]: 2017-09-11 18:15:50,301 - util.py[WARNING]: Running module resizefs (<module 'cloudinit.config.cc_resizefs' from '/usr/lib/python3.6/site-packages/cloudinit/config/cc_resizefs.py'>) failed
```

Running it manually shows:

```
[root@cloudhost ~]# xfs_info /dev/mapper/atomicos-root 
xfs_info: /dev/mapper/atomicos-root is not a mounted XFS filesystem
```

It looks like this change was desired: https://bugzilla.redhat.com/show_bug.cgi?id=1477192

We need to either get cloud-init updated to handle this or get xfs team to officially support this feature.


Version-Release number of selected component (if applicable):

[root@cloudhost ~]# rpm -q cloud-init xfsprogs
cloud-init-0.7.9-8.fc27.noarch
xfsprogs-4.12.0-4.fc27.x86_64

How reproducible:

Always

Steps to Reproduce:
1. Boot https://kojipkgs.fedoraproject.org/compose/branched/Fedora-27-20170910.n.0/compose/CloudImages/x86_64/images/Fedora-Atomic-27-20170910.n.0.x86_64.qcow2

Comment 1 Eric Sandeen 2017-09-11 19:45:22 UTC
Sigh, yes, xfs_growfs has always been documented as acting on a /mount point/ not a device.

We had reports of people doing "xfs_growfs /mount/point" where they forgot that they had unmounted /mount/point and then accidentally ended up growing the filesystem above it, i.e. the root fs.  And without shrink, they were stuck.  Hence the change.

Rejecting directories that aren't mountpoints but still allowing device paths would be a reasonable option, I guess.  too bad that an undocumented behavior ended up coded into cloud-init.  :(

Comment 2 Dusty Mabe 2017-09-11 19:57:27 UTC
(In reply to Eric Sandeen from comment #1)
> Sigh, yes, xfs_growfs has always been documented as acting on a /mount
> point/ not a device.

That is interesting. I guess I would have thought device would have been supported even before mount point. i.e. usually the progression for resizing filesystems was offline resize support and then online resize support. Hence block device as an argument is not an unreasonable thing to expect to work.

I'm not sure how things progressed in XFS but I guess online grow was always supported since the documentation only says specifying a mount point works. 

> 
> We had reports of people doing "xfs_growfs /mount/point" where they forgot
> that they had unmounted /mount/point and then accidentally ended up growing
> the filesystem above it, i.e. the root fs.  And without shrink, they were
> stuck.  Hence the change.

Yeah I understand that, tricky problem. /me would love shrink-ability on XFS. I definitely miss it from ext4. Combining XFS + lvm-thin is just not as straightforward for a normal user to comprehend. 

> 
> Rejecting directories that aren't mountpoints but still allowing device
> paths would be a reasonable option, I guess.  too bad that an undocumented
> behavior ended up coded into cloud-init.  :(

+1, I think accepting a block device (with a filesystem on top) as an argument is something people would expect to work. Rejecting directories that aren't mountpoints but still allowing device paths would prevent the problem you originally described, but allow people to still pass in block devices.

How do you feel about accepting that as an RFE?

Comment 3 Eric Sandeen 2017-09-11 20:21:23 UTC
xfs_growfs has always been exclusively an online operation, so implying that a device works in general isn't desired; it /must/ be mounted.  There is no offline resize.

If we accept (when we accepted ...) a device, it simply found the mount point from it, and used that under the covers.

What information are you gathering from xfs_info?

Comment 4 Dusty Mabe 2017-09-11 20:26:59 UTC
(In reply to Eric Sandeen from comment #3)
> xfs_growfs has always been exclusively an online operation, so implying that
> a device works in general isn't desired; it /must/ be mounted.  There is no
> offline resize.

good to know

> 
> If we accept (when we accepted ...) a device, it simply found the mount
> point from it, and used that under the covers.
> 
> What information are you gathering from xfs_info?

i'm not sure what cloud-init is doing under the covers, but I think it simply just tries to run `xfs_growfs /dev/mapper/atomicos-root` and it fails. I don't think it uses `xfs_info` at all. I assume the same root cause is what is behind xfs_info (BZ1477192). Is that a valid assumption?

Comment 5 Eric Sandeen 2017-09-11 20:29:40 UTC
Oh, so it really does want to grow. .. this bug refers to both actions, so wasn't sure...

Ok, let me huddle with my xfs dev compatriots and figure out where to go from here...

Comment 6 Dusty Mabe 2017-09-11 20:37:30 UTC
(In reply to Eric Sandeen from comment #5)
> Oh, so it really does want to grow. .. this bug refers to both actions, so
> wasn't sure...

oops - I meant to copy the output from the xfs_growfs command in the original bug description. Either way they both "fail" the same way:

```
[root@cloudhost ~]# xfs_info /dev/mapper/atomicos-root 
xfs_info: /dev/mapper/atomicos-root is not a mounted XFS filesystem
[root@cloudhost ~]# xfs_growfs /dev/mapper/atomicos-root 
xfs_growfs: /dev/mapper/atomicos-root is not a mounted XFS filesystem
```

> 
> Ok, let me huddle with my xfs dev compatriots and figure out where to go
> from here...

Thanks! I'm hoping to solve this problem for Fedora 27 so I appreciate you getting back to me so soon.

Comment 7 Eric Sandeen 2017-09-11 20:58:53 UTC
(I see that yes, cloud-init really does want to grow the filesystem not just get info w/ xfs_info).

And looking further, I think that this is all that's needed for cloud-init to invoke xfs_growfs properly:



diff --git a/cloudinit/config/cc_resizefs.py b/cloudinit/config/cc_resizefs.py
index ceee952..d78e17f 100644
--- a/cloudinit/config/cc_resizefs.py
+++ b/cloudinit/config/cc_resizefs.py
@@ -54,7 +54,7 @@ def _resize_ext(mount_point, devpth):
 
 
 def _resize_xfs(mount_point, devpth):
-    return ('xfs_growfs', devpth)
+    return ('xfs_growfs', mount_point)
 
 
 def _resize_ufs(mount_point, devpth):

Comment 8 Fedora Blocker Bugs Application 2017-09-12 13:16:18 UTC
Proposed as a Freeze Exception for 27-beta by Fedora user dustymabe using the blocker tracking app because:

 This is mostly a cosmetic bug for Atomic Host. For the cloud base image we use ext4 so we aren't affected there. It would be nice to have cloud-init working properly for xfs filesystems; proposing as freeze exception.

Comment 9 Dusty Mabe 2017-09-12 18:38:39 UTC
(In reply to Eric Sandeen from comment #7)
>
>  def _resize_xfs(mount_point, devpth):
> -    return ('xfs_growfs', devpth)
> +    return ('xfs_growfs', mount_point)
>  

seems to work. we'll try to propose to upstream cloud-init

Comment 10 Eric Sandeen 2017-09-12 18:41:11 UTC
feel free to cc: me if you like, and/or simply quote the man page ;)

Comment 11 Dusty Mabe 2017-09-13 19:15:03 UTC
upstream pull request: https://code.launchpad.net/~dustymabe/cloud-init/+git/cloud-init/+merge/330701

Comment 12 Kamil Páral 2017-09-18 17:09:32 UTC
Discussed at blocker review meeting [1]:

AcceptedFreezeException - This seems to be small change which will fix cloud-init for xfs filesystems

[1] https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2017-09-18

Comment 13 Fedora Update System 2017-09-18 22:37:35 UTC
cloud-init-0.7.9-9.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-62a8a2b453

Comment 14 Fedora Update System 2017-09-18 22:37:53 UTC
cloud-init-0.7.9-9.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-5507234819

Comment 15 Fedora Update System 2017-09-18 22:38:25 UTC
cloud-init-0.7.9-9.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-d1613e2c23

Comment 16 Kamil Páral 2017-09-19 13:21:32 UTC
(In reply to Fedora Update System from comment #15)
> cloud-init-0.7.9-9.fc27 has been submitted as an update to Fedora 27.
> https://bodhi.fedoraproject.org/updates/FEDORA-2017-d1613e2c23

I you want to see this pushed to Beta, please make sure it's tested (either here or in bodhi). Thanks.

Comment 17 Dusty Mabe 2017-09-19 21:13:26 UTC
(In reply to Kamil Páral from comment #16)
> (In reply to Fedora Update System from comment #15)
> > cloud-init-0.7.9-9.fc27 has been submitted as an update to Fedora 27.
> > https://bodhi.fedoraproject.org/updates/FEDORA-2017-d1613e2c23
> 
> I you want to see this pushed to Beta, please make sure it's tested (either
> here or in bodhi). Thanks.

I added karma in bodhi (the update is locked right now though). Please test using the following images and also add karma in bodhi:


    https://dustymabe.fedorapeople.org/cloud-init-0.7.9-9-atomic.qcow2
    https://dustymabe.fedorapeople.org/cloud-init-0.7.9-9.qcow2

9272e54c6303112f03be4dfa5dc06bf2  cloud-init-0.7.9-9-atomic.qcow2
3f850a752b59434c4fed6ebbd99bc3d5  cloud-init-0.7.9-9.qcow2

Comment 18 Fedora Update System 2017-09-19 23:28:50 UTC
cloud-init-0.7.9-9.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-d1613e2c23

Comment 19 Fedora Update System 2017-09-20 00:23:13 UTC
cloud-init-0.7.9-9.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-62a8a2b453

Comment 20 Fedora Update System 2017-09-20 01:21:38 UTC
cloud-init-0.7.9-9.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-5507234819

Comment 21 Fedora Update System 2017-09-25 18:05:49 UTC
cloud-init-0.7.9-9.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.