Bug 1891543

Summary: OpenShift 4.6/OSP install fails when node flavor has less than 25GB, even with dedicated storage
Product: OpenShift Container Platform Reporter: Ben Pritchett <bpritche>
Component: InstallerAssignee: Emilien Macchi <emacchi>
Installer sub component: OpenShift on OpenStack QA Contact: weiwei jiang <wjiang>
Status: CLOSED ERRATA Docs Contact:
Severity: urgent    
Priority: urgent CC: andbartl, blpowers, bpritche, bretm, dahernan, egarcia, emacchi, igreen, mtleilia, pprinett, wjiang
Version: 4.7Keywords: UpcomingSprint
Target Milestone: ---   
Target Release: 4.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Installer checks that the minimum 25GB disk is available per node. However, this validation only checks the osp flavor, and does not validate if separate rootDisk has been attached from dedicated storage. Consequence: When using a small flavor in combination with sufficient rootDisk, Installer refuses to install. Fix: With this patch, the additional rootDisk is taken into consideration when validating the required disk space. Result: A successful install can be carried out using a rootDisk in combination with a flavor with a small disk.
Story Points: ---
Clone Of:
: 1899161 (view as bug list) Environment:
Last Closed: 2021-02-24 15:28:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1899161    
Attachments:
Description Flags
example install-config
none
flavor combinations test report none

Description Ben Pritchett 2020-10-26 15:06:38 UTC
Created attachment 1724186 [details]
example install-config

Thanks for opening a bug report!
Before hitting the button, please fill in as much of the template below as you can.
If you leave out information, it's harder to help you.
Be ready for follow-up questions, and please respond in a timely manner.
If we can't reproduce a bug we might close your issue.
If we're wrong, PLEASE feel free to reopen it and explain why.

Version:

$ openshift-install version
$ ./openshift-install-pre version
./openshift-install-pre 4.6.0-0.nightly-2020-10-03-051134
built from commit bff124c9941762d2532490774b3f910241bd63f6
release image quay.io/openshift-release-dev/ocp-release-nightly@sha256:9663f178a9a5bf87fad0d4e2dabeaef32110d4c9c3d400eededd4f6bff5109fc

Platform:

#Please specify the platform type: aws, libvirt, openstack or baremetal etc.
openstack

Please specify:
* IPI (automated install with `openshift-install`. If you don't know, then it's IPI)
* UPI (semi-manual installation on customized infrastructure)
IPI

What happened?

With OCP 4.6 and the IPI install on OpenStack, it looks like there are new validations added to check necessary disk/memory/compute preinstall. One of the checks is to ensure the minimum 25GB disk is available per node. However, this validation only checks the osp flavor, and does not validate if separate rootDisk has been attached from dedicated storage. This means if a osp flavor is used with less than 25GB on ephemeral storage, the installer will error out, even if dedicated rootDisk is requested for more than 25GB.

#See the troubleshooting documentation (https://github.com/openshift/installer/blob/master/docs/user/troubleshooting.md) for ideas about what information to collect.

$ cat .openshift_install.log 
time="2020-10-26T14:54:50Z" level=debug msg="OpenShift Installer 4.6.0-0.nightly-2020-10-03-051134"
time="2020-10-26T14:54:50Z" level=debug msg="Built from commit bff124c9941762d2532490774b3f910241bd63f6"
time="2020-10-26T14:54:50Z" level=debug msg="Fetching Metadata..."
time="2020-10-26T14:54:50Z" level=debug msg="Loading Metadata..."
time="2020-10-26T14:54:50Z" level=debug msg="  Loading Cluster ID..."
time="2020-10-26T14:54:50Z" level=debug msg="    Loading Install Config..."
time="2020-10-26T14:54:50Z" level=debug msg="      Loading SSH Key..."
time="2020-10-26T14:54:50Z" level=debug msg="      Loading Base Domain..."
time="2020-10-26T14:54:50Z" level=debug msg="        Loading Platform..."
time="2020-10-26T14:54:50Z" level=debug msg="      Loading Cluster Name..."
time="2020-10-26T14:54:50Z" level=debug msg="        Loading Base Domain..."
time="2020-10-26T14:54:50Z" level=debug msg="        Loading Platform..."
time="2020-10-26T14:54:50Z" level=debug msg="      Loading Pull Secret..."
time="2020-10-26T14:54:50Z" level=debug msg="      Loading Platform..."
time="2020-10-26T14:54:52Z" level=fatal msg="failed to fetch Metadata: failed to load asset \"Install Config\": platform.openstack.flavorName: Invalid value: \"osp.2xlarge\": Flavor did not meet the following minimum requirements: Must have minimum of 25 GB Disk, had 20 GB"


What did you expect to happen?

With attached dedicated storage, the validation for 25GB minimum disk is not checked only on the osp flavor.

How to reproduce it (as minimally and precisely as possible)?

$ openshift-install create cluster --dir=my-cluster/ --log-level=debug
In this case the install-config.yaml references a osp flavor with less than 25GB, but has rootDisk attributed to the node from dedicated storage offering (see example install-config.yaml)

Comment 8 weiwei jiang 2020-11-10 07:42:33 UTC
Checked with 
$ ./openshift-install-4.7 version 
./openshift-install-4.7 4.7.0-0.nightly-2020-11-09-190845
built from commit 519f21bfada103752bb78929f170580d505b92c0
release image registry.svc.ci.openshift.org/ocp/release@sha256:f7bf10489f11c05927e901310ad7d824599b385f69f12c2de61cf8c437c85fc4

and found that rootVolume do not take the first priority to decide the failure.


apiVersion: v1           
baseDomain: 10.0.103.240.nip.io
compute:   
- architecture: amd64
  hyperthreading: Enabled
  name: worker     
  platform:       
    rootVolume:
      size: 25
      type: tripleo  
    type: ci.usm-qe-02   
  replicas: 2 
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master     
  platform:                                                                                                                                                                                                                                                                     
    rootVolume:
      size: 25
      type: tripleo 
    type: ci.usm-qe-02
  replicas: 3    
metadata:              
  name: wj45ios1019a
networking:      
  clusterNetwork:       
  - cidr: 10.128.0.0/14    
    hostPrefix: 23
  machineNetwork:
  - cidr: 192.168.0.0/18
  networkType: OpenShiftSDN
  serviceNetwork: 
  - 172.30.0.0/16              
platform:                              
  openstack:                  
    cloud: upshift     
    computeFlavor: ci.usm-qe-02
    externalNetwork: provider_net_cci_8
    lbFloatingIP: 10.0.103.240
    octaviaSupport: '0'
    region: regionOne                                                                                                                                                                                                                                                           
    trunkSupport: '1'                                                                                                                   
publish: External        
pullSecret: Hidden                                                                                                                      
sshKey: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCWkwurd8TNAi+D7ffvyDdhGBSQtJx3/Yedlwvvha0q772vLlOAGlKCw4dajKy6qty1/GGQDgTJ17h3C9TEArI8ZqILnyydeY56DL+ELN3dtGBVof/N2qtW0+SmEnd1Mi7Qy5Tx4e/GVmB3NgX9szwNOVXhebzgBsXc9x+RtCVLPLC8J+qqSdTUZ0UfJsh2ptlQLGHmmTpF//QlJ1tngvAFeCOxJUhrLAa3
7P9MtFsiNk31EfKyBk3eIdZljTERmqFaoJCohsFFEdO7tVgU6p5NwniAyBGZVjZBzjELoI1aZ+/g9yReIScxl1R6PWqEzcU6lGo2hInnb6nuZFGb+90D
  openshift-qe
Running: ./openshift-install-4.7 create manifests --dir /tmp/tmp.cbKWELvH3Y
FATAL failed to fetch Master Machines: failed to load asset "Install Config": platform.openstack.computeFlavor: Invalid value: "ci.usm-qe-02": Flavor did not meet the following minimum requirements: Must have minimum of 25 GB Disk, had 20 GB

Comment 12 weiwei jiang 2020-11-11 06:34:20 UTC
Created attachment 1728265 [details]
flavor combinations test report

Comment 32 David Hernández Fernández 2020-12-08 09:46:30 UTC
For reference, backport to 4.6.z: https://bugzilla.redhat.com/show_bug.cgi?id=1899161  | https://github.com/openshift/installer/pull/4394

Comment 35 egarcia 2021-02-04 20:12:39 UTC
What disk size does the image you are using have? Could we also see the install config?

Comment 36 Emilien Macchi 2021-02-04 20:50:24 UTC
also please provide details on the flavor m6.xlarge.8 (specs).

Comment 41 errata-xmlrpc 2021-02-24 15:28:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633

Comment 43 Pierre Prinetti 2021-11-11 07:48:24 UTC
Blake,
Please create a new BZ detailing:
* the Installer version
* the error output 
* a redacted copy-paste of your install-config
* the output of `openstack flavor show ${flavor}` for the master or the worker flavor, depending on what's causing problem

Thank you!