Description : Spot instance not getting created on AzureGovCloud Version :Cluster version is 4.6.0-0.nightly-2020-09-23-022756 m/m Reproducible :Always Steps : 1.refer below yaml to create spot instance machineset , make sure to provide higher price at that moment , try google for latest price for the type of instance mentioned in machineset https://gist.github.com/miyadav/ca5f85e78ff0cd98f7ddabb85607f110 Expected : machineset should be created successfully 2.oc get machines Expected :machine should be provisioned successfully Actual : machine creation failed [miyadav@miyadav ManualRun]$ oc get machines NAME PHASE TYPE REGION ZONE AGE miyadav-az-zzlxl-master-0 Running Standard_D8s_v3 usgovvirginia 44h miyadav-az-zzlxl-master-1 Running Standard_D8s_v3 usgovvirginia 44h miyadav-az-zzlxl-master-2 Running Standard_D8s_v3 usgovvirginia 44h miyadav-az-zzlxl-worker-spot-cx9hs Failed 5s miyadav-az-zzlxl-worker-usgovvirginia-nd8gh Running Standard_D2s_v3 usgovvirginia 3h7m miyadav-az-zzlxl-worker-usgovvirginia-qxmbr Running Standard_D2s_v3 usgovvirginia 20h miyadav-az-zzlxl-worker-usgovvirginia-xdnjd Running Standard_D2s_v3 usgovvirginia 20h Additional log : I0925 05:53:12.606654 1 machine_scope.go:160] miyadav-az-zzlxl-worker-spot-hjxrs: patching machine E0925 05:53:12.687288 1 actuator.go:78] Machine error: failed to reconcile machine "miyadav-az-zzlxl-worker-spot-hjxrs": failed to create vm miyadav-az-zzlxl-worker-spot-hjxrs: failure sending request for machine miyadav-az-zzlxl-worker-spot-hjxrs W0925 05:53:12.687320 1 controller.go:315] miyadav-az-zzlxl-worker-spot-hjxrs: failed to create machine: failed to reconcile machine "miyadav-az-zzlxl-worker-spot-hjxrs": failed to create vm miyadav-az-zzlxl-worker-spot-hjxrs: failure sending request for machine miyadav-az-zzlxl-worker-spot-hjxrs I0925 05:53:12.687329 1 controller.go:415] Actuator returned invalid configuration error: failed to reconcile machine "miyadav-az-zzlxl-worker-spot-hjxrs": failed to create vm miyadav-az-zzlxl-worker-spot-hjxrs: failure sending request for machine miyadav-az-zzlxl-worker-spot-hjxrs I0925 05:53:12.687339 1 controller.go:424] miyadav-az-zzlxl-worker-spot-hjxrs: going into phase "Failed" I0925 05:53:12.687543 1 recorder.go:52] controller-runtime/manager/events "msg"="Warning" "message"="InvalidConfiguration: failed to reconcile machine \"miyadav-az-zzlxl-worker-spot-hjxrs\": failed to create vm miyadav-az-zzlxl-worker-spot-hjxrs: failure sending request for machine miyadav-az-zzlxl-worker-spot-hjxrs" "object"={"kind":"Machine","namespace":"openshift-machine-api","name":"miyadav-az-zzlxl-worker-spot-hjxrs","uid":"33ce149a-d069-4d43-9423-6f5a3289a91e","apiVersion":"machine.openshift.io/v1beta1","resourceVersion":"2274306"} "reason"="FailedCreate"
@Milind, I don't think I have access to GovCloud to try to reproduce this, could you try again but without setting a maxPrice, just use `spotVMOptions: {}` instead? Would be good to know if it is related to any Spot VM or just if there is a maxPrice
@joel - tried as you suggested : providerSpec: value: spotVMOptions: {} Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreate 18s azure-controller InvalidConfiguration: failed to reconcile machine "miyadav-az-zzlxl-worker-spot-fzd4r": failed to create vm miyadav-az-zzlxl-worker-spot-fzd4r: failure sending request for machine miyadav-az-zzlxl-worker-spot-fzd4r
I'm aware that on AWS, their GovCloud didn't support spot instances for a very long time, though I believe it does now. I wonder if Azure just doesn't support creating spot instances on the GovCloud tenant. I think we will need to do some research and try to work out if they should be supported on GovCloud or not.
I'm not able to launch any spot instance using Azure GovCloud console. I think spot instances are not supported in GovCloud.
I spoke with Alex yesterday who confirmed (as per https://bugzilla.redhat.com/show_bug.cgi?id=1882608#c4) that when trying to create an instance in Azure GovCloud, independently of OCP, that the same error occurs. Since we are aware that spot instances can be created in other regions, I suspect that the issue here is that Azure does not support Spot Instances in this region. I've asked Alex to try and confirm if this is the case and hopefully we will have an answer to this question during the next sprint. If this is the case, we may be able to improve the error message that we return to improve UX. Additionally we should ensure that the docs team are aware and can add a note that this region doesn't support spot (if that is the case).
Spot instances are most likely not supported on GovCloud.
I've reached out to some folks from Microsoft to try and get an answer as to whether these are supported or not. My suggestion is that we add a warning via a webhook if we can see that the region is GovCloud and Spot VMs are requested, for example "Warning: Spot VMs may not be supported in this region". I will update this issue if I get a response.
@ademicev I've had no response from MS about this. I think the route to closing this issue should be adding a webhook warning when spot instances are requested in the GovCloud region, something akin to "Warning: Spot VMs may not be supported in this region". Could you create a PR to add that please?
the PR associated with this has all the necessary labels and is currently waiting on CI to pass
Validated on: [miyadav@miyadav ~]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.0-0.ci-2020-12-08-014807 True False 30m Cluster version is 4.7.0-0.ci-2020-12-08-014807 Steps: 1.Create machineset to provision spot instance , refer below . . spec: metadata: {} providerSpec: value: spotVMOptions: maxPrice: 0.225 . . Actual & Expected - machineset created successfully 2.Monitor logs ( oc logs -f machine-controller-pod -c machine-controller Actual & Expected - I1208 05:03:35.631580 1 actuator.go:168] Updating machine miyadav608az-zvd4x-worker-usgovvirginia-s-v9tlq I1208 05:03:35.894049 1 machine_scope.go:144] miyadav608az-zvd4x-worker-usgovvirginia-s-v9tlq: status unchanged I1208 05:03:35.894084 1 machine_scope.go:144] miyadav608az-zvd4x-worker-usgovvirginia-s-v9tlq: status unchanged I1208 05:03:35.894089 1 machine_scope.go:160] miyadav608az-zvd4x-worker-usgovvirginia-s-v9tlq: patching machine W1208 05:03:35.907643 1 warnings.go:67] spot VMs may not be supported when using GovCloud region Additional info: Moved to Verified
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633