Bug 1497062 - Node 4.1.4 and later wont install on 35GB disk.
Summary: Node 4.1.4 and later wont install on 35GB disk.
Keywords:
Status: CLOSED CANTFIX
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-node-ng
Version: 4.1.5
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: ---
Assignee: Ryan Barry
QA Contact: Qin Yuan
URL:
Whiteboard:
Depends On:
Blocks: ovirt-node-ng-43-el76-platform
TreeView+ depends on / blocked
 
Reported: 2017-09-29 06:53 UTC by Germano Veit Michel
Modified: 2020-08-03 15:34 UTC (History)
15 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-06-19 09:47:34 UTC
oVirt Team: Node
Target Upstream Version:
lsvaty: testing_plan_complete-


Attachments (Terms of Use)
anaconda-tb (421.73 KB, text/plain)
2018-03-29 08:31 UTC, Samantha N. Bueno
no flags Details
anaconda install log rhvh 4.1 20180410 on 43Gb disk (476.68 KB, text/plain)
2018-04-18 19:38 UTC, Andrea Perotti
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1374007 None CLOSED [RFE] RHV-H does not default to LVM Thin Provisioning 2019-09-12 20:19:04 UTC
Red Hat Bugzilla 1569249 None CLOSED [Docs][Planning] New minimum disk size for RHV-H since 4.1.4 2019-09-12 20:19:04 UTC

Internal Links: 1374007 1569249

Description Germano Veit Michel 2017-09-29 06:53:27 UTC
Description of problem:

Documentation[1] states that the minimum supported disk size for RHV-H is 35GB:

Table 2.4. Red Hat Virtualization Host Minimum Storage Requirements
Partition        Minimum Size
/                6 GB
/home            1 GB
/tmp             1 GB
/boot            1 GB
/var            15 GB
/var/log         8 GB
/var/log/audit   2 GB
swap             1 GB
Minimum Total   35 GB    <---------

Installs work fine using layout above on 35Gb disks until 4.1.3. From 4.1.4+ this is not enough anymore and the install fails with the attached error.

Newer limit seems to be around 48GB (automatic) or 44GB (manual) in order to succeed. 

Not sure if this is a Docs bug or RHV-H needs to go on a diet. But it is a problem for customers who relied on the 35GB information and now can't get 4.1.4+ installed due to not having enough disk space in the hosts.

Note 4.1.3- custom partition does not default to lvm thin, which was a bug, I think.

[1] https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.1/html/planning_and_prerequisites_guide/requirements

Version-Release number of selected component (if applicable):
RHV-H 4.1.3 and later.

How reproducible:
100%

Steps to Reproduce:
1. Try to install 4.1.4+ on a 35GB disk following the docs table.

Comment 3 Ryan Barry 2017-09-30 00:16:24 UTC
The custom partition layout issue was an Anaconda bug fixed in 7.4, but I can't think of a reason why the required space for rhvh would have changed. I'll test this

Comment 4 Ryan Barry 2017-10-02 13:30:00 UTC
This is definitely reproducible. I wonder if there are some new storage constraints in 7.4

I'll check. We probably need a docs update.

Comment 5 Ryan Barry 2017-10-06 16:56:35 UTC
This definitely appears to be related to constraints in Anaconda, since the size of RHVH has not changed significantly.

Samantha -- any idea why this is happening? Our installclass doesn't require anywhere near this much space:

https://github.com/rhinstaller/anaconda/blob/rhel7-branch/pyanaconda/installclasses/rhv.py

Comment 6 cshao 2017-10-09 08:40:28 UTC
QE can reproduce this issue on RHVH 4.1.4 (redhat-virtualization-host-4.1-20170808.0).

Test steps:
1. Try to install 4.1.4+ on a 35GB disk.
2. Focus on install process.

Test result:
No space left on device.

Comment 8 David Sundqvist 2017-10-19 10:51:08 UTC
Ran in to this as well. Keeping a minimum size target of around 32GB at least within reach has some advantages, unless there are actual reasons for the size growth.

For example, small devices such as USB sticks are a compelling installation target for Node for various use cases, and they often come in a 32GB size, so I can imagine there are a fair number of users that the growth would cause issues for.

Comment 9 Germano Veit Michel 2018-01-12 04:26:34 UTC
Ryan are there any known workarounds at this point?

Comment 10 Ryan Barry 2018-01-12 13:14:10 UTC
Using the anaconda environment from 4.1.3 (or any 7.3 version) is a viable workaround until 7.5, when platform should have a fix for this.

Comment 11 Samantha N. Bueno 2018-02-01 15:09:07 UTC
Can we have installation logs from a failed attempt? (Note: I did check the logs in the customer ticket, but they appear to show a successful installation.)

Comment 12 Ryan Barry 2018-02-01 15:14:26 UTC
Not that I know of, but this should be trivially reproducible by grabbing a RHVH ISO from the portal and using a small disk in a vm

Comment 13 Germano Veit Michel 2018-02-02 06:04:41 UTC
(In reply to Samantha N. Bueno from comment #11)
> Can we have installation logs from a failed attempt? (Note: I did check the
> logs in the customer ticket, but they appear to show a successful
> installation.)

Hi Samantha, this is easily reproducible by downloading[1] the ISO and attempting to install it on a Virtual Machine with a 35GB Disk in your PC. I assume it would be easier to troubleshoot in the live system.

But if you somehow cant do I can grab the logs for you, or leave a failed install ready somewhere. What files exactly do you want me to grab for you? Do I need to enable any extra debug?

[1] Hypervisor Image for RHV 4.1.9 
https://access.redhat.com/downloads/content/150/ver=4.1/rhel---7/4.1/x86_64/product-software

Comment 14 Sandro Bonazzola 2018-02-15 16:29:22 UTC
Samantha do we have an anaconda bug for this we can track?
Can you add it to the dependencies of this bug?

Comment 15 Samantha N. Bueno 2018-03-27 15:00:42 UTC
The attached screenshot shows anaconda failing on a %pre script, so this is something that is failing pretty early on in installation.

I ventured in to look at the customer case. The customer logs that are attached there seem to show a successful, completed installation. The customer also attached a screenshot of the error window, but that image is at direct odds with the logs they attached, so I have to assume that image and the logs are from two separate installation attempts.

I don't know what special handling is going on in kickstart, but from the traceback it seems like there is something going on. Could you please attach the kickstart file being used? And logs genuinely would be helpful. We can try to reproduce, but without the ks file or a set of specific steps to follow, I don't have high confidence that the logs we'd generate would be a faithful reproduction of the issue.

As for why this is happening in 7.4 v 7.3 -- I'm not really sure. I looked over the 7.4 changelog for anaconda, and we certainly did introduce changes related to constraints, but I could spot nothing that should have an effect like this. So that means I missed something; there are unintended consequences from another change at play here; something is perhaps a little misconfigured in a ks file; or some combination.

Comment 16 David Lehman 2018-03-27 15:24:34 UTC
As per recommendation from the LVM team, anaconda reserves 20% of the thin pool size within the volume group for future metadata expansion. This is to prevent an out-of-the-box configuration from running out of space under normal usage conditions. Overprovisioning of thin pools during installation is also not supported, so your only option here is going to be to either use a larger disk or reduce the size of some of the thin logical volumes. My guess is that if you did not see this limitation before you were not using thin provisioning.

Comment 17 Ryan Barry 2018-03-27 15:30:13 UTC
RHVH 4.x has always used thinpools.

Prior to 7.4, we did not see this limitation.

Is it possible to override this?

Comment 18 Ryan Barry 2018-03-27 15:30:13 UTC
RHVH 4.x has always used thinpools.

Prior to 7.4, we did not see this limitation.

Is it possible to override this?

Comment 19 Samantha N. Bueno 2018-03-27 16:44:09 UTC
Ok, I'll have to eat my words and say this is pretty simple to reproduce. But, I don't even get to a point where I can begin an installation and give it a chance to fail. Not if I hit the traceback attached in comment 1.

That is from something else entirely. It is from this bit of a %pre kickstart script that is automatically getting run (I didn't dig too deep into the RHVH iso but I assume something there is causing this to run automatically):

> cd /tmp
> rpm2cpio /run/install/repo/Packages/redhat-virtualization-host-image-update*|cpio -ivd
> squashfs=$(find|grep squashfs|grep -v meta)
> ln -s $squashfs /tmp/squashfs

I'm presuming this is so that you can theme/brand anaconda with RHVH instead of RHEL, based on the packages that are being unpacked here. But this is what is causing the issue -- or rather, the way you are doing this is causing the issue. /tmp is filling up, hence the "No space left on device" error.

Have you tried running through the customization guide to implement these customizations? Did you run into issues, or is the amount of customization available to you not enough?

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/anaconda_customization_guide/index#sect-iso-images

Comment 20 Ryan Barry 2018-03-27 17:49:51 UTC
The VM must have at least 2GB 

We actually have a working branding package which is not related to that %pre snippet. redhat-virtualization-host-productimg.

The %pre bit is there (written and maintained by RCM) because RHVH is shipped as a squashfs packed inside an RPM, and this way RCM doesn't need to change it for every NVR.

Interactive Anaconda can't use a liveimg as a repo if selected, but it can and will install from one if it's prefilled as a spoke, which is what this does.

Have you tried a machine with 2gb+ of memory?

Comment 22 Samantha N. Bueno 2018-03-29 08:30:01 UTC
(In reply to Ryan Barry from comment #20)
> The VM must have at least 2GB 
> 
> We actually have a working branding package which is not related to that
> %pre snippet. redhat-virtualization-host-productimg.
> 
> The %pre bit is there (written and maintained by RCM) because RHVH is
> shipped as a squashfs packed inside an RPM, and this way RCM doesn't need to
> change it for every NVR.
> 
> Interactive Anaconda can't use a liveimg as a repo if selected, but it can
> and will install from one if it's prefilled as a spoke, which is what this
> does.
> 
> Have you tried a machine with 2gb+ of memory?

Ok, I see, thanks for explaining. So the attachment in comment 1 is not the actual issue at hand then, since that's what I see when I attempt this on a box with not enough memory.

Now I assume (on a box with enough memory) that I do see the actual error. I tried this on a significantly smaller disk since I assumed the behavior would be the same. I got a traceback, which I'm hoping is the same  experience of others on this report -- that was never explicitly stated.

So I will attach that traceback, and this might be where I ask someone from python-blivet to look in.

Comment 23 Samantha N. Bueno 2018-03-29 08:31:58 UTC
Created attachment 1414641 [details]
anaconda-tb

Traceback generated from installing on a disk that's too small.

Comment 24 Ryan Barry 2018-04-11 12:27:21 UTC
Deferring, since this did not make 7.5

Comment 25 Andrea Perotti 2018-04-18 19:38:07 UTC
Created attachment 1423712 [details]
anaconda install log rhvh 4.1 20180410 on 43Gb disk

Comment 26 Andrea Perotti 2018-04-18 20:19:21 UTC
attached my anaconda log when attempting to install RHVH-4.1 2018-04-10.x86_64 in  KVM on a 43G disk using default partitioning.

Manually creating each LV thin allow you to install it on a 35 disk again, but some partitions are really small.

Using the default use the size suggested as minimum by our documentation:
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2-beta/html/planning_and_prerequisites_guide/requirements#storage_requirements

but you need 45Gb at least: 1G for boot + 44Gb for vg rhvh with 7.82Gb free.

For sure, while waiting for this to be fixed, and the target is now months ahead, we should update our doc from 4.1 to avoid any misunderstanding with our customer base, and this change probably would be good to be documente in the 4.2 release notes as well.

Comment 31 Franta Kust 2019-05-16 13:05:22 UTC
BZ<2>Jira Resync


Note You need to log in before you can comment on or make changes to this bug.