Bug 1925343

Summary: [ci] e2e-metal tests are not using reserved instances
Product: OpenShift Container Platform Reporter: Matthew Staebler <mstaeble>
Component: InstallerAssignee: Matthew Staebler <mstaeble>
Installer sub component: openshift-installer QA Contact: Gaoyun Pei <gpei>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high    
Version: 4.7   
Target Milestone: ---   
Target Release: 4.8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-07-27 22:41:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Matthew Staebler 2021-02-04 22:24:03 UTC
The e2e-metal tests should be using the instances in the ny5 datacenter that we have reserved. Currently, the tests are running on any datacenter and on any available hardware.

Comment 1 Scott Dodson 2021-02-05 13:19:57 UTC
Here's an example commit from when I tested reserved instances in the past, I know I mentioned this on slack but in case someone other than Matthew ends up working on this bug here it is.

https://github.com/openshift/installer/commit/66c3fee9731116ed70c1a9fcad3ad6eaa4dc905b

I believe in order to do what we're asked we need to change the facilities to ['ny5'], the plan to 'c3.small.x86', and hardware_reservation_id = "next-available"

There's only 39 (weird number?) reserved instances, so with 7 hosts per cluster (1 bootstrap, 3 control-plane, 3 worker) that only leaves enough for five clusters. We don't currently tear down the bootstrap node once it's done, maybe we should fix that. If we were that would probably enable us to run six concurrent clusters in this reserved pool rather than five. We may also want to look at boskos leases just to ensure that we don't attempt to start more than we want at once.

The terraform code in the installer repo was meant to be generic enough that it could be used outside of our specific CI jobs and that's why most of these are variables so it's worth considering whether that's worth splitting the config between installer and release repo.

Comment 2 Scott Dodson 2021-02-05 13:20:48 UTC
Oh, I also don't believe this is urgent so feel free to lower that to high/high if you wish.

Comment 4 Matthew Staebler 2021-04-19 17:23:06 UTC
We'll look to progress further on this next sprint.

Comment 7 Gaoyun Pei 2021-06-13 06:50:36 UTC
This is a CI issue for adding packet_facility, packet_plan and packet_hardware_reservation_id variables to the terraform templates, so we could use the reserved instances in e2e-metal CI testing.

Checked the e2e-metal-4.8 test in latest two payloads: 4.8.0-0.nightly-2021-06-12-174011, 4.8.0-0.nightly-2021-06-12-223426, both succeeded as below, move this bug as VERIFIED.

https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-metal-4.8/1403844375636611072
https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-metal-4.8/1403769632917032960

Comment 10 errata-xmlrpc 2021-07-27 22:41:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438