Bug 1561888 - ovirt-hosted-engine-setup and cockpit-ovirt do not agree about the minimum size for HE storage [NEEDINFO]
Summary: ovirt-hosted-engine-setup and cockpit-ovirt do not agree about the minimum si...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: rhevm-appliance
Version: 2.1.0
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ovirt-4.2.2
: ---
Assignee: Ryan Barry
QA Contact: Nikolai Sednev
URL:
Whiteboard:
Depends On:
Blocks: 1522737
TreeView+ depends on / blocked
 
Reported: 2018-03-29 06:32 UTC by Yihui Zhao
Modified: 2019-05-16 13:08 UTC (History)
18 users (show)

Fixed In Version: rhvm-appliance-4.2-20180404.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-05-15 19:00:03 UTC
oVirt Team: Node
stirabos: needinfo? (rstory)


Attachments (Terms of Use)
He deploy in cockpit (58.84 KB, image/png)
2018-04-05 13:46 UTC, Pavol Brilla
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:1525 None None None 2018-05-15 19:02:30 UTC

Description Yihui Zhao 2018-03-29 06:32:30 UTC
Description of problem: 
When deploying HE via cockpit, cockpit limits the VM disk size : 58~4096G
But from the CLI, the default vm size is 50G.
"""
[ INFO  ] TASK [Check storage domain free space]
[ INFO  ] skipping: [localhost]
          Please specify the size of the VM disk in GB: [50]: 
[ INFO  ] Creating Target VM
[ INFO  ] TASK [Gathering Facts]
"""


Version-Release number of selected component (if applicable): 
rhvh-4.2.2.0-0.20180322.0+1
cockpit-ovirt-dashboard-0.11.19-1.el7ev.noarch
ovirt-hosted-engine-setup-2.2.14-1.el7ev.noarch
ovirt-hosted-engine-ha-2.2.7-1.el7ev.noarch
rhvm-appliance-4.2-20180322.0.el7.noarch


How reproducible: 


Steps to Reproduce: 
1. Deploy HE with NFS storage via cockpit
 
Actual results:  
The same as the description.

Expected results: 
The default VM size from cockpit should be the same as CLI

Additional info:

Comment 1 Ryan Barry 2018-03-29 12:01:05 UTC
Simone -

The appliance actually uses 58gb since 4.1. Are we sure 50gb is correct?

Comment 2 Simone Tiraboschi 2018-03-29 14:10:57 UTC
(In reply to Ryan Barry from comment #1)
> Simone -
> 
> The appliance actually uses 58gb since 4.1. Are we sure 50gb is correct?

AFAIK is 50

Comment 5 Pavol Brilla 2018-04-05 13:46:17 UTC
Created attachment 1417723 [details]
He deploy in cockpit

Comment 6 Pavol Brilla 2018-04-05 13:47:21 UTC
Failing as I cannot decrease size of disk below 58G, see attached screenshot

tested with: rhvm-appliance-4.2-20180404.0

Comment 7 Ryan Barry 2018-04-05 15:21:22 UTC
This is exactly the point, Pavol. After adding swap, 58gb is correct.

The OVA was updated. Please try a CLI deploy withe same appliance, and it should now also reflect 58gb

Comment 8 Nikolai Sednev 2018-04-09 11:18:27 UTC
This is the message that being received during deployment over 70GB iSCSI LUN from Cockpit:
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Error: the target storage domain contains only 64.0GiB of available space while a minimum of 68.0GiB is required"}

And this is what I've got from CLI for the same 70GB LUN:
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Error: the target storage domain contains only 64.0GiB of available space while a minimum of 68.0GiB is required"}

Deployment over NFS on 58GB from Cockpit was also accepted now.

Tested on:
cockpit-ovirt-dashboard-0.11.20-1.el7ev.noarch
ovirt-hosted-engine-setup-2.2.16-1.el7ev.noarch
ovirt-hosted-engine-ha-2.2.10-1.el7ev.noarch
rhvm-appliance-4.2-20180404.0.el7.noarch
Linux 3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.5 (Maipo)

Works for me the same for both CLI and Cockpit, during deployment phase.

Ryan, please provide your input about this data:
The thing here is that "a minimum of 68.0GiB is required", does not take in account the 6GB of metadata, hence, IMHO, message should properly inform customers, that deployment requires 6GB for metadata and minimum of 68GB for the deployment, hence actual minimal LUN space should be at least 74GB (68+6=74). 
I think that message here can be refined, so customers will provide proper LUN space. Just in case, I've tested deployment with 74GB of raw LUN's space and deployment proceeded farther, so 74GB of raw LUN's space is the minimum for the deployment over iSCSI.

Comment 9 Ryan Barry 2018-04-09 12:35:04 UTC
I think there are two things we can do here, Nikolai, but both happen on the ovirt-hosted-engine-side.

The first is to update the documentation and the message to essentially be ${size_of_image}+10% (to account for metadata)

The other, which we probably don't want to do, is to remove the metadata reserved space on the LV.

Simone, can we update the message to add 10%?

Comment 10 Simone Tiraboschi 2018-04-09 13:17:29 UTC
(In reply to Ryan Barry from comment #9)
> Simone, can we update the message to add 10%?

This is not that accurate.
On a 2TB we are not going to have a 200Gb overhead.

We are creating a SD from the engine and it adds some overhead here but we cannot really forecast how much on hosted-engine-setup side.
We are simply reporting to the user the free space after SD creation.

Comment 12 Nikolai Sednev 2018-04-22 07:35:16 UTC
Moving to verified again, forth to my previous comment #8.

Comment 13 Robert Story 2018-05-09 15:22:23 UTC
(In reply to Nikolai Sednev from comment #8)
> Tested on:
> cockpit-ovirt-dashboard-0.11.20-1.el7ev.noarch
> ovirt-hosted-engine-setup-2.2.16-1.el7ev.noarch
> ovirt-hosted-engine-ha-2.2.10-1.el7ev.noarch
> rhvm-appliance-4.2-20180404.0.el7.noarch
> Linux 3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64
> x86_64 x86_64 GNU/Linux

I just tried 75GB on CentOS 7.4 and it failed:

ovirt-hosted-engine-setup-2.2.20-1.el7.centos.noarch
ovirt-engine-appliance-4.2-20180504.1.el7.centos.noarch

> Ryan, please provide your input about this data:
> The thing here is that "a minimum of 68.0GiB is required", does not take in
> account the 6GB of metadata, hence, IMHO, message should properly inform
> customers, that deployment requires 6GB for metadata and minimum of 68GB for
> the deployment, hence actual minimal LUN space should be at least 74GB
> (68+6=74). 
> I think that message here can be refined, so customers will provide proper
> LUN space. Just in case, I've tested deployment with 74GB of raw LUN's space
> and deployment proceeded farther, so 74GB of raw LUN's space is the minimum
> for the deployment over iSCSI.

75 doesn't seem to be enough:

2018-05-09 10:57:30,942-0400 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:98 fatal: [localhost]: FAILED! => {"ansible_facts": {"ovirt_storage_domains": [{"available": 17179869184, "backup": false, "committed": 83751862272, "critical_space_action_blocker": 5, "data_centers": [{"href": "/ovirt-engine/api/datacenters/9b63311a-5396-11e8-90ff-00163e42c5f0", "id": "9b63311a-5396-11e8-90ff-00163e42c5f0"}], "discard_after_delete": false, "disk_profiles": [{"id": ["4345bdc3-7556-4770-9033-36a83edfb850"], "name": "hosted_storage"}], "disk_snapshots": [], "disks": [{"id": ["d2126176-9a73-4f0e-9b33-1ef9a0783a19"], "image_id": "b502cca7-1e39-41b0-a055-2991afce6c9b", "name": "he_sanlock"}, {"id": ["99a6dc02-dca1-46df-846d-23a828a08719"], "image_id": "9f41662a-cb4a-4fe9-a64e-f1415ef938f2", "name": "he_virtio_disk"}, {"id": ["d2eeba63-ade5-44d4-aec8-fe15722ed704"], "image_id": "0aac3f5f-7ada-42a8-a4ca-741d99bcd34b", "name": "HostedEngineConfigurationImage"}, {"id": ["ee8edf28-f336-4231-8c34-cc71fd49b536"], "image_id": "9a534c16-f34c-4f6b-9fb5-fe069508adc3", "name": "he_metadata"}], "external_status": "ok", "href": "/ovirt-engine/api/storagedomains/e20a2d75-2a8d-4475-999f-6c2ad2afd8c4", "id": "e20a2d75-2a8d-4475-999f-6c2ad2afd8c4", "master": true, "name": "hosted_storage", "permissions": [{"id": ["54158708-5397-11e8-8ad8-00163e42c5f0"]}, {"id": ["58ca605c-010d-0307-0224-0000000001a9"]}], "storage": {"type": "iscsi", "volume_group": {"id": "zjlmQF-1DcZ-3DUu-kC7Z-20B2-Dhs2-LHPf4k", "logical_units": [{"address": "10.1.0.2", "discard_max_size": 0, "discard_zeroes_data": false, "id": "360014050330d6cfd86aad4d31d8107d2", "lun_mapping": 0, "paths": 0, "port": 3260, "portal": "10.1.0.2:3260,1", "product_id": "iSCSI Storage", "serial": "SSYNOLOGYiSCSI_Storage_0330d6cf-86aa-4d31-8107-2bf60679f706", "size": 107374182400, "storage_domain_id": "e20a2d75-2a8d-4475-999f-6c2ad2afd8c4", "target": "iqn.2000-01.com.synology:ov.ov-engine.48518fc5b7", "vendor_id": "SYNOLOGY", "volume_group_id": "zjlmQF-1DcZ-3DUu-kC7Z-20B2-Dhs2-LHPf4k"}]}}, "storage_connections": [{"id": ["58da59e4-835d-420e-8563-2a96cc634e14"]}], "storage_format": "v4", "supports_discard": false, "supports_discard_zeroes_data": false, "templates": [], "type": "data", "used": 89120571392, "vms": [], "warning_low_space_indicator": 10, "wipe_after_delete": false}]}, "attempts": 12, "changed": false}

Comment 16 Simone Tiraboschi 2018-05-09 19:28:08 UTC
"size": 107374182400, "used": 89120571392 means that it failed for Robert on a 100 GiB LUN having already used 83 GiB before trying to create HE disk.

Robert, can you please attach vdsm.log and engine.log?

Comment 17 Robert Story 2018-05-10 21:12:03 UTC
Hmm.. I thought I had nuked the LVM volumes between installs. Unfortunately I don't have logs, as I've already tried 2 more installs since then, and am in the middle of a third. For most recent installs it turned out to be networking issues, and I was using a larger VM size (82G).

Comment 18 Robert Story 2018-05-11 02:57:09 UTC
(In reply to Nikolai Sednev from comment #8)
> Ryan, please provide your input about this data:
> The thing here is that "a minimum of 68.0GiB is required", does not take in
> account the 6GB of metadata, hence, IMHO, message should properly inform
> customers, that deployment requires 6GB for metadata and minimum of 68GB for
> the deployment, hence actual minimal LUN space should be at least 74GB
> (68+6=74). 

So I finally have an install up and running, and according to my math, overhead is right around 8GB, not 6. So 76 is a safer bet.

I had a 100GB iscsi target, and I specified 80GB for my VM.

# pvscan
  PV /dev/mapper/360...7d2   VG b55...db7   lvm2 [99.62 GiB / 11.50 GiB free]

99.6 - 11.6 - 80 = 8

The details on LVs:

# lvscan 
/dev/b55...b7/metadata' [512.00 MiB] inherit
/dev/b55...b7/outbox' [128.00 MiB] inherit
/dev/b55...b7/xleases' [1.00 GiB] inherit
/dev/b55...b7/leases' [2.00 GiB] inherit
/dev/b55...b7/ids' [128.00 MiB] inherit
/dev/b55...b7/inbox' [128.00 MiB] inherit
/dev/b55...b7/master' [1.00 GiB] inherit
/dev/b55...b7/c070a4d0-2896-4763-8ef9-88eccac3f593' [80.00 GiB] inherit
/dev/b55...b7/08a71ffe-4e65-4096-ba8b-c1998cfa5f1b' [1.00 GiB] inherit
/dev/b55...b7/8b0c284f-3d04-42f9-a639-aaa217c8ab0c' [1.00 GiB] inherit
/dev/b55...b7/d6870255-c213-451c-aa70-f24ee7644edb' [1.00 GiB] inherit
/dev/b55...b7/2df29878-eb04-40f8-82bc-befd566ae8e9' [128.00 MiB] inherit
/dev/b55...b7/fd4c7583-c1f1-417e-94c4-f330b615b813' [128.00 MiB] inherit

Comment 19 Robert Story 2018-05-11 02:58:42 UTC
oops, slight error in my math: 99.62 - 11.5 - 80 = 8.12

Comment 20 Nikolai Sednev 2018-05-13 17:18:58 UTC
I've just successfully deployed over FC using 74GiB LUN:
         Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs)[nfs]: fc
[ INFO  ] Getting Fibre Channel LUNs list
[ INFO  ] TASK [Gathering Facts]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [include_tasks]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [Obtain SSO token using username/password credentials]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [Get Fibre Channel LUNs]
[ INFO  ] ok: [localhost]
          The following luns have been found on the requested target:
                [1]     3514f0c5a51601776       74GiB   XtremIO XtremApp
                        status: free, paths: 2 active
         
          Please select the destination LUN (1) [1]: 
[ INFO  ] FC discard after delete is enabled
[ INFO  ] Creating Storage Domain
.
.
.
[ INFO  ] changed: [localhost]
          Please specify the size of the VM disk in GB: [58]: 
[ INFO  ] Creating Target VM
.
.
.
[ INFO  ] Stage: Termination
[ INFO  ] Hosted Engine successfully deployed

I've tested on downstream using these components:
ovirt-engine-setup-4.2.3.4-0.1.el7.noarch
rhvm-appliance-4.2-20180504.0.el7.noarch
ovirt-hosted-engine-ha-2.2.11-1.el7ev.noarch
ovirt-hosted-engine-setup-2.2.20-1.el7ev.noarch
Linux 3.10.0-862.2.3.el7.x86_64 #1 SMP Mon Apr 30 12:37:51 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.5 (Maipo)

Comment 21 Nikolai Sednev 2018-05-14 05:43:47 UTC
Same deployment over iSCSI using 74GiB clean LUN also was successful:
[ INFO  ] changed: [localhost]
          Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs)[nfs]: iscsi
          Please specify the iSCSI portal IP address: 10.35.146.129
          Please specify the iSCSI portal port [3260]: 
          Please specify the iSCSI discover user: 
          Please specify the iSCSI discover password: 
          Please specify the iSCSI portal login user: 
          Please specify the iSCSI portal login password: 
[ INFO  ] Discovering iSCSI targets
[ INFO  ] TASK [Gathering Facts]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [include_tasks]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [Obtain SSO token using username/password credentials]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [Prepare iSCSI parameters]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [Fetch host facts]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [iSCSI discover with REST API]
[ INFO  ] ok: [localhost]
          The following targets have been found:
                [1]     iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f6c05
                        TPGT: 1, portals:
                                10.35.146.225:3260
         
                [2]     iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f6c04
                        TPGT: 1, portals:
                                10.35.146.193:3260
         
                [3]     iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f6c01
                        TPGT: 1, portals:
                                10.35.146.161:3260
         
                [4]     iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f6c00
                        TPGT: 1, portals:
                                10.35.146.129:3260
         
          Please select a target (1, 2, 3, 4) [1]: 
[ INFO  ] Getting iSCSI LUNs list
[ INFO  ] TASK [Gathering Facts]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [include_tasks]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [Obtain SSO token using username/password credentials]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [iSCSI login]
[ INFO  ] TASK [Get iSCSI LUNs]
[ INFO  ] ok: [localhost]
          The following luns have been found on the requested target:
                [1]     3514f0c5a51601777       74GiB   XtremIO XtremApp
                        status: free, paths: 1 active
         
                [2]     3514f0c5a51601778       100GiB  XtremIO XtremApp
                        status: free, paths: 1 active
         
          Please select the destination LUN (1, 2) [1]: 
[ INFO  ] iSCSI discard after delete is enabled
.
.
.
[ INFO  ] iSCSI connected paths: 1
          Please specify the size of the VM disk in GB: [58]: 
[ INFO  ] Creating Target VM
.
.
.
[ INFO  ] Stage: Termination
[ INFO  ] Hosted Engine successfully deployed

I've tested on downstream using these components:
ovirt-engine-setup-4.2.3.4-0.1.el7.noarch
rhvm-appliance-4.2-20180504.0.el7.noarch
ovirt-hosted-engine-ha-2.2.11-1.el7ev.noarch
ovirt-hosted-engine-setup-2.2.20-1.el7ev.noarch
Linux 3.10.0-862.2.3.el7.x86_64 #1 SMP Mon Apr 30 12:37:51 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.5 (Maipo)

Comment 22 Simone Tiraboschi 2018-05-14 07:28:36 UTC
AFAIK we have:
6 GiB: SD overhead
58 GiB: default HE VM disk
3 GiB: HE ancillary disks (configuration, metadata and lockspace)
2 GiB: OVF_STOREs
5 GiB: Critical space action blocker

So 74 GiB is the minimum allowed size.

Comment 24 Robert Story 2018-05-14 21:25:14 UTC
TLDR summary: I agree, maybe.

There is some confusion here with units. install asks for engine size in GB, but creates the lv in GiB. My NAS vendor does the same thing.  I created what I thought was a 74GB volume, but fdisk on the connected LUN reports 79.5 GB. 

After the install, lvm2 report [73.62 GiB / 7.50 GiB free], so 74 GiB not only seems to be enough, but more than enough (unless some temporary lvs for 7.5 GiB are created and deleted during the process).


I'll file a separate bug for the incorrect units in the install question.

Comment 25 Nikolai Sednev 2018-05-15 03:45:21 UTC
1GB(Gigabyte)=10^9=1 000 000 000 000 in decimal value.
1GiB(Gibibyte)=2^30=1 073 741 824 in binary value.
So 74GB=68.9179GiB

My storage vendor and also in FreeNAS provide this in GB as for storage units and I've used 74 decimal value.

When referring to computer memory, gigabyte is always a “power of two” - 1,073,741,824 bytes, but when measuring hard drive capacity it is often defined as 1,000,000,000 bytes. OS calculates disk and file sizes using binary numbers, so a new 500 GB drive you've just purchased would be reported by the OS as "465.66 GB" (meaning 465.66 GiB).

I'm not sure where is a bug here exactly.

Comment 28 errata-xmlrpc 2018-05-15 19:00:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:1525

Comment 29 Franta Kust 2019-05-16 13:08:47 UTC
BZ<2>Jira Resync


Note You need to log in before you can comment on or make changes to this bug.