Bug 1878900 - Installer complains about not enough vcpu for the baremetal flavor where generic bm flavor is being used
Summary: Installer complains about not enough vcpu for the baremetal flavor where gene...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.6
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ---
: 4.7.0
Assignee: Pierre Prinetti
QA Contact: weiwei jiang
URL:
Whiteboard:
Depends On:
Blocks: 1889416
TreeView+ depends on / blocked
 
Reported: 2020-09-14 20:16 UTC by Chris Janiszewski
Modified: 2021-02-24 15:18 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Feature: Skip flavor validation for baremetal flavors Reason: Sometimes, OpenStack administrators do not set the appropriate metadata on their baremetal flavors. The Installer pre-flight validation, which is performed on the flavor metadata, might then prevent the installation whereas the actual BM would have all the required capacity. Result: With this patch, validations are skipped on flavors detected as "baremetal".
Clone Of:
Environment:
Last Closed: 2021-02-24 15:17:46 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 4222 0 None closed Bug 1878900: openstack: Skip validation for baremetal flavors 2021-02-15 19:53:50 UTC
Github openshift installer pull 4269 0 None closed Bug 1878900: openstack: Fix error messages in flavor validation 2021-02-15 19:53:49 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:18:22 UTC

Description Chris Janiszewski 2020-09-14 20:16:30 UTC
Description of problem:

Installer complains about not enough vcpu for the baremetal flavor where generic bm flavor is being used

(shiftstack) [stack@chrisj-undercloud-osp13 ~]$ openshift-install --dir=ocpra create cluster --log-level=debug                                                                
DEBUG OpenShift Installer 4.6.0-0.nightly-2020-09-10-145837 
DEBUG Built from commit 1d3e041bcc50dee31138bb911da7e5fd4b101f7f 
DEBUG Fetching Metadata...                         
DEBUG Loading Metadata...                          
DEBUG   Loading Cluster ID...                      
DEBUG     Loading Install Config...                
DEBUG       Loading SSH Key...                     
DEBUG       Loading Base Domain...                 
DEBUG         Loading Platform...                  
DEBUG       Loading Cluster Name...                
DEBUG         Loading Base Domain...               
DEBUG         Loading Platform...                  
DEBUG       Loading Pull Secret...                 
DEBUG       Loading Platform...                    
FATAL failed to fetch Metadata: failed to load asset "Install Config": compute[0].platform.openstack.flavorName: Invalid value: "baremetal": Flavor did not meet the following minimum requirements: Must have minimum of 2 VCPUs, had 1 

when deploying on OpenStack Ironic nodes, it's not unusual to have a generic small baremetal flavor that catches all type of hardware including ones that hasn't been properly inspected.

This must be a new check on OCP 4.6 since I have not been getting that in OCP 4.5. It probably makes sense to not have that check with Baremetal nodes



Version-Release number of the following components:
OpenShift Installer 4.6.0-0.nightly-2020-09-10-145837

Comment 6 weiwei jiang 2020-10-12 02:14:36 UTC
Checked again, and got following results:

$ ./openshift-install version 
./openshift-install 4.6.0-0.nightly-2020-10-10-041109
built from commit ebdbda57fc18d3b73e69f0f2cc499ddfca7e6593
release image registry.svc.ci.openshift.org/ocp/release@sha256:a33eff631dd29cfe412d9cdcfaec68844a18b902e657708e16dbd85203da4eed


# #install-config.yaml with only compute as baremetal:
---
apiVersion: v1
baseDomain: wjiang.shiftstack.io
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  platform:
    openstack:
      flavorName: baremetal
  replicas: 2
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  platform: {}
  replicas: 3
platform:
  openstack:
    apiVIP: 192.168.0.5
    cloud: shiftstack
    computeFlavor: m4.xlarge
    externalDNS: null
    externalNetwork: public
    ingressVIP: 192.168.0.7
    lbFloatingIP: 10.46.43.177
...

$ ./openshift-install-4.6 create cluster --dir .
INFO Consuming Install Config from target directory 
INFO Credentials loaded from file "/home/stack/wjiang/clouds.yaml" 
INFO Obtaining RHCOS image file from 'https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/rhcos-4.6/46.82.202009222340-0/x86_64/rhcos-46.82.202009222340-0-openstack.x86_64.qcow2.gz?sha256=cc8c4be1bd1c537c5b236a217901f9cd708a15b05010410d74e63632304a121e' 
INFO The file was found in cache: /home/stack/.cache/openshift-installer/image_cache/766103797521c4c4bd93293d43a5b025. Reusing... 
INFO Creating infrastructure resources...         
INFO Waiting up to 20m0s for the Kubernetes API at https://api.bmocp.wjiang.shiftstack.io:6443... 
...


# #install-config.yaml with both compute and control-plane as baremetal:
---
apiVersion: v1
baseDomain: wjiang.shiftstack.io
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  platform:
    openstack:
      flavorName: baremetal
  replicas: 2
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  platform:
    openstack:
      flavorName: baremetal
  replicas: 3
platform:
  openstack:
    apiVIP: 192.168.0.5
    cloud: shiftstack
    computeFlavor: m4.xlarge
    externalDNS: null
    externalNetwork: public
    ingressVIP: 192.168.0.7
    lbFloatingIP: 10.46.43.177
...

$ ./openshift-install-4.6 create cluster --dir .
INFO Consuming Install Config from target directory 
INFO Credentials loaded from file "/home/stack/wjiang/clouds.yaml" 
INFO Obtaining RHCOS image file from 'https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/rhcos-4.6/46.82.202009222340-0/x86_64/rhcos-46.82.202009222340-0-openstack.x86_64.qcow2.gz?sha256=cc8c4be1bd1c537c5b236a217901f9cd708a15b05010410d74e63632304a121e' 
INFO The file was found in cache: /home/stack/.cache/openshift-installer/image_cache/766103797521c4c4bd93293d43a5b025. Reusing... 
INFO Creating infrastructure resources...         
INFO Waiting up to 20m0s for the Kubernetes API at https://api.bmocp.wjiang.shiftstack.io:6443...

# #install-config.yaml with all flavor as baremetal:
---
apiVersion: v1
baseDomain: wjiang.shiftstack.io
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  platform:
    openstack:
      flavorName: baremetal
  replicas: 2
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  platform:
    openstack:
      flavorName: baremetal
  replicas: 3
...
platform:
  openstack:
    apiVIP: 192.168.0.5
    cloud: shiftstack
    computeFlavor: baremetal
    externalDNS: null
    externalNetwork: public
    ingressVIP: 192.168.0.7
    lbFloatingIP: 10.46.43.177
...


$ ./openshift-install-4.6 create cluster --dir . --log-level debug
DEBUG OpenShift Installer 4.6.0-0.nightly-2020-09-27-075304 
DEBUG Built from commit 527e3beb3609be50551b4019282ece14cce1ecce 
DEBUG Fetching Metadata...                         
DEBUG Loading Metadata...                          
DEBUG   Loading Cluster ID...                      
DEBUG     Loading Install Config...                
DEBUG       Loading SSH Key...                     
DEBUG       Loading Base Domain...                 
DEBUG         Loading Platform...                  
DEBUG       Loading Cluster Name...                
DEBUG         Loading Base Domain...               
DEBUG         Loading Platform...                  
DEBUG       Loading Pull Secret...                 
DEBUG       Loading Platform...                    
FATAL failed to fetch Metadata: failed to load asset "Install Config": platform.openstack.flavorName: Invalid value: "baremetal": Flavor did not meet the following minimum requirements: Must have minimum of 4 VCPUs, had 2; Must have minimum of 25 GB Disk, had 10 GB 

So do that means we only allow flavorName in compute and controlPlane as baremetal?

Comment 7 weiwei jiang 2020-10-12 02:19:35 UTC
Add the flavor info back.
$ openstack flavor show baremetal
+----------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
| Field                      | Value                                                                                                                                 |
+----------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
| OS-FLV-DISABLED:disabled   | False                                                                                                                                 |
| OS-FLV-EXT-DATA:ephemeral  | 0                                                                                                                                     |
| access_project_ids         | None                                                                                                                                  |
| description                | None                                                                                                                                  |
| disk                       | 10                                                                                                                                    |
| extra_specs                | {'baremetal': 'true', 'resources:VCPU': '0', 'resources:MEMORY_MB': '0', 'resources:DISK_GB': '0', 'resources:CUSTOM_BAREMETAL': '1'} |
| id                         | 15d0f90c-8022-4202-ba88-dd7f9bea151e                                                                                                  |
| name                       | baremetal                                                                                                                             |
| os-flavor-access:is_public | True                                                                                                                                  |
| properties                 | baremetal='true', resources:CUSTOM_BAREMETAL='1', resources:DISK_GB='0', resources:MEMORY_MB='0', resources:VCPU='0'                  |
| ram                        | 4096                                                                                                                                  |
| rxtx_factor                | 1.0                                                                                                                                   |
| swap                       | 0                                                                                                                                     |
| vcpus                      | 2                                                                                                                                     |
+----------------------------+---------------------------------------------------------------------------------------------------------------------------------------+

Comment 9 Martin André 2020-10-14 07:48:26 UTC
(In reply to weiwei jiang from comment #6)
> # #install-config.yaml with all flavor as baremetal:
> ---
> apiVersion: v1
> baseDomain: wjiang.shiftstack.io
> compute:
> - architecture: amd64
>   hyperthreading: Enabled
>   name: worker
>   platform:
>     openstack:
>       flavorName: baremetal

Hey Weiwei, just a note that flavorName is not a valid property for the openstack machine pool. It should be `mp.platform.openstack.type`, see https://github.com/openshift/installer/blob/master/docs/user/openstack/customization.md#machine-pools.
So your machine pools will inherit the flavors from platform.openstack.computeFlavor. But that does not explain why validation is still complaining when setting platform.openstack.computeFlavor to the baremetal flavor.

Comment 13 Pierre Prinetti 2020-10-16 12:51:52 UTC
The latest patch brings a refactoring that should make the behaviour more consistent. Additionally, the error messages have been fixed to reflect the actual name of the property in install-config.

Comment 16 Pierre Prinetti 2020-11-03 08:21:33 UTC
@Chris
The validation of baremetal flavours should not happen anymore in master. Did you have the chance to check if the fix was effective for you?

Comment 20 weiwei jiang 2020-12-28 02:54:52 UTC
Checked with OpenShift Installer 4.7.0-0.nightly-2020-12-17-201522 and it works well now.


[root@openshift-qe-001 ~]# openstack flavor show baremetal
+----------------------------+----------------------------------------------------------------------------------------------------------------------+
| Field                      | Value                                                                                                                |
+----------------------------+----------------------------------------------------------------------------------------------------------------------+                                                                                                                           
| OS-FLV-DISABLED:disabled   | False                                                                                                                |                                                                                                                           
| OS-FLV-EXT-DATA:ephemeral  | 0                                                                                                                    |                                                                                                                           
| access_project_ids         | None                                                                                                                 |                                                                                                                           
| disk                       | 39                                                                                                                   |                                                                                                                           
| id                         | e7f810f7-31c5-42ee-b486-b166f5208090                                                                                 |                                                                                                                           
| name                       | baremetal                                                                                                            |                                                                                                                           
| os-flavor-access:is_public | True                                                                                                                 |                                                                                                                           
| properties                 | baremetal='true', resources:CUSTOM_BAREMETAL='1', resources:DISK_GB='0', resources:MEMORY_MB='0', resources:VCPU='0' |                                                                                                                           
| ram                        | 4096                                                                                                                 |                                                                                                                           
| rxtx_factor                | 1.0                                                                                                                  |                                                                                                                           
| swap                       |                                                                                                                      |                                                                                                                           
| vcpus                      | 2                                                                                                                    |                                                                                                                           
+----------------------------+----------------------------------------------------------------------------------------------------------------------+ 


# cat install-config.yaml
apiVersion: v1
baseDomain: 10.0.103.130.nip.io
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  platform:
    openstack:
      type: baremetal
  replicas: 2
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  platform:
    openstack:
      type: baremetal
  replicas: 3
metadata:
  name: wj45ios1019a
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineNetwork:
  - cidr: 192.168.0.0/18
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16
platform:
  openstack:
    cloud: shiftstack
    externalNetwork: nova
    computeFlavor: baremetal
publish: External
pullSecret: 'HIDDEN'

# ./openshift-install-4.7 create manifests --dir .
INFO Credentials loaded from file "/etc/openstack/clouds.yaml" 
INFO Consuming Install Config from target directory 
INFO Manifests created in: manifests and openshift

Comment 21 Chris Janiszewski 2021-01-19 17:26:41 UTC
sorry Pierre by super late response. I am still deploying 4.6 and currently at 4.6.12 which still shows this issue:

DEBUG OpenShift Installer 4.6.12                   
DEBUG Built from commit eded5eb5b6c77e2af2a2c537093da8bf3711f494 
DEBUG Fetching Metadata...                         
DEBUG Loading Metadata...                          
DEBUG   Loading Cluster ID...                      
DEBUG     Loading Install Config...                
DEBUG       Loading SSH Key...                     
DEBUG       Loading Base Domain...                 
DEBUG         Loading Platform...                  
DEBUG       Loading Cluster Name...                
DEBUG         Loading Base Domain...               
DEBUG         Loading Platform...                  
DEBUG       Loading Pull Secret...                 
DEBUG       Loading Platform...                    
FATAL failed to fetch Metadata: failed to load asset "Install Config": compute[0].platform.openstack.type: Invalid value: "baremetal": Flavor did not meet the following minimum requirements: Must have minimum of 2 VCPUs, had 1 


I am not ready to test 4.7, but it looks like someone else has validated this above. Thanks for working on this

Comment 22 Pierre Prinetti 2021-01-19 17:47:37 UTC
Thank you Chris for checking!

The fix is definitely merged in 4.6.12 (which is built from commit eded5eb5b6c77e2af2a2c537093da8bf3711f494) with Bug 1889416.

If the fix is not working for you, it might mean that the marker we use to identify baremetal flavors is not universal.

The Installer currently suspends flavor checks when the flavor has one extra spec property "baremetal" with value "true"[1].

Would you be able to check what marker can be used in your environment to assess whether a flavor is baremetal? (ideally, something that can be seen with `openstack flavor show $baremetal_flavor`)

Thank you!


[1]: https://github.com/openshift/installer/pull/4222/files#diff-49aba5b26cc5cdc067b5e54eb6595f1fd820389055010f96b7764bae301c1cc1R160-R173

Comment 23 Chris Janiszewski 2021-01-19 18:06:04 UTC
Good call .. indeed I don't have that property in my flavor and I agree that most deployment would have it. So let's consider this a configuration issue.

(hextupleo) [stack@undercloud-hextupleo ~]$ openstack flavor show baremetal
+----------------------------+------------------------------------------------------------------------------------------------------------------+
| Field                      | Value                                                                                                            |
+----------------------------+------------------------------------------------------------------------------------------------------------------+
| OS-FLV-DISABLED:disabled   | False                                                                                                            |
| OS-FLV-EXT-DATA:ephemeral  | 0                                                                                                                |
| access_project_ids         | None                                                                                                             |
| description                | None                                                                                                             |
| disk                       | 40                                                                                                               |
| extra_specs                | {'resources:CUSTOM_BAREMETAL': '1', 'resources:VCPU': '0', 'resources:MEMORY_MB': '0', 'resources:DISK_GB': '0'} |
| id                         | 11280c38-df3e-415f-8ff2-f2be074c4868                                                                             |
| name                       | baremetal                                                                                                        |
| os-flavor-access:is_public | True                                                                                                             |
| properties                 | resources:CUSTOM_BAREMETAL='1', resources:DISK_GB='0', resources:MEMORY_MB='0', resources:VCPU='0'               |
| ram                        | 1024                                                                                                             |
| rxtx_factor                | 1.0                                                                                                              |
| swap                       | 0                                                                                                                |
| vcpus                      | 1                                                                                                                |
+----------------------------+------------------------------------------------------------------------------------------------------------------+


for me the other properties make the trick to target a BM nodes rather then VM instances, hence I have not had baremetal=true
Please disregard then. Thanks again!

Comment 25 errata-xmlrpc 2021-02-24 15:17:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.