Bug 1770128 - [Azure] Machine does not become 'Failed' when it has create error
Summary: [Azure] Machine does not become 'Failed' when it has create error
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cloud Compute
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.4.0
Assignee: Alberto
QA Contact: Jianwei Hou
URL:
Whiteboard:
Depends On:
Blocks: 1776759
TreeView+ depends on / blocked
 
Reported: 2019-11-08 09:05 UTC by Jianwei Hou
Modified: 2020-05-15 14:43 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1776759 (view as bug list)
Environment:
Last Closed: 2020-05-15 14:43:36 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-api-provider-azure pull 91 0 'None' closed bug 1770128: UPSTREAM: <carry>: openshift: Fix statusCode conversion 2020-12-09 06:08:10 UTC

Description Jianwei Hou 2019-11-08 09:05:27 UTC
Description of problem:
With an invalid providerSpec create a machine(for example set vmSize: Standard_D2s_v3_badsize), the machine stays at 'Provisioning' phase where it is expected to be 'Failed'.

Version-Release number of selected component (if applicable):
4.3.0-0.nightly-2019-11-07-172437

How reproducible:
Always

Steps to Reproduce:
1. on Azure IPI, edit a machineset and inject an invalid input, for example, set an invalid vmSize.
2. Scale up the machineset
3. Retrieve the machine status

Actual results:
Machine is stuck at 'Provisioning' phase for 29 minutes.

jhou-dbjpx-worker-centralus2-8gqs5   Provisioning                                        29m

oc describe machine jhou-dbjpx-worker-centralus2-8gqs5
Name:         jhou-dbjpx-worker-centralus2-8gqs5
Namespace:    openshift-machine-api
Labels:       machine.openshift.io/cluster-api-cluster=jhou-dbjpx
              machine.openshift.io/cluster-api-machine-role=worker
              machine.openshift.io/cluster-api-machine-type=worker
              machine.openshift.io/cluster-api-machineset=jhou-dbjpx-worker-centralus2
Annotations:  <none>
API Version:  machine.openshift.io/v1beta1
Kind:         Machine
Metadata:
  Creation Timestamp:  2019-11-08T08:29:46Z
  Finalizers:
    machine.machine.openshift.io
  Generate Name:  jhou-dbjpx-worker-centralus2-
  Generation:     2
  Owner References:
    API Version:           machine.openshift.io/v1beta1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  MachineSet
    Name:                  jhou-dbjpx-worker-centralus2
    UID:                   23f05a9c-c565-4f23-9c6a-f95edcbb75d7
  Resource Version:        156736
  Self Link:               /apis/machine.openshift.io/v1beta1/namespaces/openshift-machine-api/machines/jhou-dbjpx-worker-centralus2-8gqs5
  UID:                     e80d7ccd-cd7c-485c-8df7-57914f1560bf
Spec:
  Metadata:
    Creation Timestamp:  <nil>
  Provider Spec:
    Value:
      API Version:  azureproviderconfig.openshift.io/v1beta1
      Credentials Secret:
        Name:       azure-cloud-credentials
        Namespace:  openshift-machine-api
      Image:
        Offer:
        Publisher:
        Resource ID:     /resourceGroups/jhou-dbjpx-rg/providers/Microsoft.Compute/images/jhou-dbjpx
        Sku:
        Version:
      Kind:              AzureMachineProviderSpec
      Location:          centralus
      Managed Identity:  jhou-dbjpx-identity
      Metadata:
        Creation Timestamp:    <nil>
      Network Resource Group:  jhou-dbjpx-rg
      Os Disk:
        Disk Size GB:  128
        Managed Disk:
          Storage Account Type:  Premium_LRS
        Os Type:                 Linux
      Public IP:                 false
      Resource Group:            jhou-dbjpx-rg
      Subnet:                    jhou-dbjpx-worker-subnet
      User Data Secret:
        Name:   worker-user-data
      Vm Size:  Standard_D2s_v3_badsize
      Vnet:     jhou-dbjpx-vnet
      Zone:     2
Status:
  Last Updated:  2019-11-08T08:29:49Z
  Phase:         Provisioning
  Provider Status:
    Conditions:
      Last Probe Time:       2019-11-08T08:29:49Z
      Last Transition Time:  2019-11-08T08:29:49Z
      Message:               failed to create vm jhou-dbjpx-worker-centralus2-8gqs5 : failed to create or get machine: cannot create vm: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidParameter" Message="The requested VM size Standard_D2s_v3_badsize is not available in the current region. The sizes available in the current region are: Standard_A0,Standard_A1,Standard_A2,Standard_A3,Standard_A5,Standard_A4,Standard_A6,Standard_A7,Basic_A0,Basic_A1,Basic_A2,Basic_A3,Basic_A4,Standard_D1_v2,Standard_D2_v2,Standard_D3_v2,Standard_D4_v2,Standard_D5_v2,Standard_D11_v2,Standard_D12_v2,Standard_D13_v2,Standard_D14_v2,Standard_D15_v2,Standard_D2_v2_Promo,Standard_D3_v2_Promo,Standard_D4_v2_Promo,Standard_D5_v2_Promo,Standard_D11_v2_Promo,Standard_D12_v2_Promo,Standard_D13_v2_Promo,Standard_D14_v2_Promo,Standard_F1,Standard_F2,Standard_F4,Standard_F8,Standard_F16,Standard_A1_v2,Standard_A2m_v2,Standard_A2_v2,Standard_A4m_v2,Standard_A4_v2,Standard_A8m_v2,Standard_A8_v2,Standard_DS1,Standard_DS2,Standard_DS3,Standard_DS4,Standard_DS11,Standard_DS12,Standard_DS13,Standard_DS14,Standard_D2_v3,Standard_D4_v3,Standard_D8_v3,Standard_D16_v3,Standard_D32_v3,Standard_D1,Standard_D2,Standard_D3,Standard_D4,Standard_D11,Standard_D12,Standard_D13,Standard_D14,Standard_F2s_v2,Standard_F4s_v2,Standard_F8s_v2,Standard_F16s_v2,Standard_F32s_v2,Standard_F48s_v2,Standard_F64s_v2,Standard_F72s_v2,Standard_B1ls,Standard_B1ms,Standard_B1s,Standard_B2ms,Standard_B2s,Standard_B4ms,Standard_B8ms,Standard_B12ms,Standard_B16ms,Standard_B20ms,Standard_DS1_v2,Standard_DS2_v2,Standard_DS3_v2,Standard_DS4_v2,Standard_DS5_v2,Standard_DS11-1_v2,Standard_DS11_v2,Standard_DS12-1_v2,Standard_DS12-2_v2,Standard_DS12_v2,Standard_DS13-2_v2,Standard_DS13-4_v2,Standard_DS13_v2,Standard_DS14-4_v2,Standard_DS14-8_v2,Standard_DS14_v2,Standard_DS2_v2_Promo,Standard_DS3_v2_Promo,Standard_DS4_v2_Promo,Standard_DS5_v2_Promo,Standard_DS11_v2_Promo,Standard_DS12_v2_Promo,Standard_DS13_v2_Promo,Standard_DS14_v2_Promo,Standard_F1s,Standard_F2s,Standard_F4s,Standard_F8s,Standard_F16s,Standard_D48_v3,Standard_D64_v3,Standard_D2s_v3,Standard_D4s_v3,Standard_D8s_v3,Standard_D16s_v3,Standard_D32s_v3,Standard_D48s_v3,Standard_D64s_v3,Standard_E2_v3,Standard_E4_v3,Standard_E8_v3,Standard_E16_v3,Standard_E20_v3,Standard_E32_v3,Standard_E48_v3,Standard_E64i_v3,Standard_E64_v3,Standard_E2s_v3,Standard_E4-2s_v3,Standard_E4s_v3,Standard_E8-2s_v3,Standard_E8-4s_v3,Standard_E8s_v3,Standard_E16-4s_v3,Standard_E16-8s_v3,Standard_E16s_v3,Standard_E20s_v3,Standard_E32-8s_v3,Standard_E32-16s_v3,Standard_E32s_v3,Standard_E48s_v3,Standard_E64-16s_v3,Standard_E64-32s_v3,Standard_E64is_v3,Standard_E64s_v3,Standard_DS15_v2,Standard_G1,Standard_G2,Standard_G3,Standard_G4,Standard_G5,Standard_GS1,Standard_GS2,Standard_GS3,Standard_GS4,Standard_GS4-4,Standard_GS4-8,Standard_GS5,Standard_GS5-8,Standard_GS5-16,Standard_L4s,Standard_L8s,Standard_L16s,Standard_L32s.\r\nFind out more on the available VM sizes in each region at https://aka.ms/azure-regions." Target="vmSize"
      Reason:                MachineCreationFailed
      Status:                True
      Type:                  MachineCreated
    Metadata:
      Creation Timestamp:  <nil>
Events:
  Type     Reason        Age                   From              Message
  ----     ------        ----                  ----              -------
  Warning  FailedCreate  4m34s (x71 over 29m)  azure-controller  CreateError: failed to reconcile machine "jhou-dbjpx-worker-centralus2-8gqs5"s: failed to create vm jhou-dbjpx-worker-centralus2-8gqs5 : failed to create or get machine: cannot create vm: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidParameter" Message="The requested VM size Standard_D2s_v3_badsize is not available in the current region. The sizes available in the current region are: Standard_A0,Standard_A1,Standard_A2,Standard_A3,Standard_A5,Standard_A4,Standard_A6,Standard_A7,Basic_A0,Basic_A1,Basic_A2,Basic_A3,Basic_A4,Standard_D1_v2,Standard_D2_v2,Standard_D3_v2,Standard_D4_v2,Standard_D5_v2,Standard_D11_v2,Standard_D12_v2,Standard_D13_v2,Standard_D14_v2,Standard_D15_v2,Standard_D2_v2_Promo,Standard_D3_v2_Promo,Standard_D4_v2_Promo,Standard_D5_v2_Promo,Standard_D11_v2_Promo,Standard_D12_v2_Promo,Standard_D13_v2_Promo,Standard_D14_v2_Promo,Standard_F1,Standard_F2,Standard_F4,Standard_F8,Standard_F16,Standard_A1_v2,Standard_A2m_v2,Standard_A2_v2,Standard_A4m_v2,Standard_A4_v2,Standard_A8m_v2,Standard_A8_v2,Standard_DS1,Standard_DS2,Standard_DS3,Standard_DS4,Standard_DS11,Standard_DS12,Standard_DS13,Standard_DS14,Standard_D2_v3,Standard_D4_v3,Standard_D8_v3,Standard_D16_v3,Standard_D32_v3,Standard_D1,Standard_D2,Standard_D3,Standard_D4,Standard_D11,Standard_D12,Standard_D13,Standard_D14,Standard_F2s_v2,Standard_F4s_v2,Standard_F8s_v2,Standard_F16s_v2,Standard_F32s_v2,Standard_F48s_v2,Standard_F64s_v2,Standard_F72s_v2,Standard_B1ls,Standard_B1ms,Standard_B1s,Standard_B2ms,Standard_B2s,Standard_B4ms,Standard_B8ms,Standard_B12ms,Standard_B16ms,Standard_B20ms,Standard_DS1_v2,Standard_DS2_v2,Standard_DS3_v2,Standard_DS4_v2,Standard_DS5_v2,Standard_DS11-1_v2,Standard_DS11_v2,Standard_DS12-1_v2,Standard_DS12-2_v2,Standard_DS12_v2,Standard_DS13-2_v2,Standard_DS13-4_v2,Standard_DS13_v2,Standard_DS14-4_v2,Standard_DS14-8_v2,Standard_DS14_v2,Standard_DS2_v2_Promo,Standard_DS3_v2_Promo,Standard_DS4_v2_Promo,Standard_DS5_v2_Promo,Standard_DS11_v2_Promo,Standard_DS12_v2_Promo,Standard_DS13_v2_Promo,Standard_DS14_v2_Promo,Standard_F1s,Standard_F2s,Standard_F4s,Standard_F8s,Standard_F16s,Standard_D48_v3,Standard_D64_v3,Standard_D2s_v3,Standard_D4s_v3,Standard_D8s_v3,Standard_D16s_v3,Standard_D32s_v3,Standard_D48s_v3,Standard_D64s_v3,Standard_E2_v3,Standard_E4_v3,Standard_E8_v3,Standard_E16_v3,Standard_E20_v3,Standard_E32_v3,Standard_E48_v3,Standard_E64i_v3,Standard_E64_v3,Standard_E2s_v3,Standard_E4-2s_v3,Standard_E4s_v3,Standard_E8-2s_v3,Standard_E8-4s_v3,Standard_E8s_v3,Standard_E16-4s_v3,Standard_E16-8s_v3,Standard_E16s_v3,Standard_E20s_v3,Standard_E32-8s_v3,Standard_E32-16s_v3,Standard_E32s_v3,Standard_E48s_v3,Standard_E64-16s_v3,Standard_E64-32s_v3,Standard_E64is_v3,Standard_E64s_v3,Standard_DS15_v2,Standard_G1,Standard_G2,Standard_G3,Standard_G4,Standard_G5,Standard_GS1,Standard_GS2,Standard_GS3,Standard_GS4,Standard_GS4-4,Standard_GS4-8,Standard_GS5,Standard_GS5-8,Standard_GS5-16,Standard_L4s,Standard_L8s,Standard_L16s,Standard_L32s.\r\nFind out more on the available VM sizes in each region at https://aka.ms/azure-regions." Target="vmSize"


Expected results:
Machine becomes 'Failed'

Additional info:

Comment 2 Jianwei Hou 2019-12-20 06:43:49 UTC
Verified in 4.4.0-0.nightly-2019-12-19-223334

Machine has 'Failed' phase when it has create error.

oc describe machine qe-jhou-hnqvw-worker-centralus1-sbhpx                 git:machine_healthcheck*
Name:         qe-jhou-hnqvw-worker-centralus1-sbhpx
Namespace:    openshift-machine-api
Labels:       machine.openshift.io/cluster-api-cluster=qe-jhou-hnqvw
              machine.openshift.io/cluster-api-machine-role=worker
              machine.openshift.io/cluster-api-machine-type=worker
              machine.openshift.io/cluster-api-machineset=qe-jhou-hnqvw-worker-centralus1
Annotations:  <none>
API Version:  machine.openshift.io/v1beta1
Kind:         Machine
Metadata:
  Creation Timestamp:  2019-12-20T06:39:01Z
  Finalizers:
    machine.machine.openshift.io
  Generate Name:  qe-jhou-hnqvw-worker-centralus1-
  Generation:     2
  Owner References:
    API Version:           machine.openshift.io/v1beta1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  MachineSet
    Name:                  qe-jhou-hnqvw-worker-centralus1
    UID:                   45f9b276-3b7b-4839-82ed-a43900cbe032
  Resource Version:        24308
  Self Link:               /apis/machine.openshift.io/v1beta1/namespaces/openshift-machine-api/machines/qe-jhou-hnqvw-worker-centralus1-sbhpx
  UID:                     b3576d0f-9922-44a2-9f24-6bbf4c76856f
Spec:
  Metadata:
    Creation Timestamp:  <nil>
  Provider Spec:
    Value:
      API Version:  azureproviderconfig.openshift.io/v1beta1
      Credentials Secret:
        Name:       azure-cloud-credentials
        Namespace:  openshift-machine-api
      Image:
        Offer:
        Publisher:
        Resource ID:     /resourceGroups/qe-jhou-hnqvw-rg/providers/Microsoft.Compute/images/qe-jhou-hnqvw
        Sku:
        Version:
      Kind:              AzureMachineProviderSpec
      Location:          centralus
      Managed Identity:  qe-jhou-hnqvw-identity
      Metadata:
        Creation Timestamp:    <nil>
      Network Resource Group:  qe-jhou-hnqvw-rg
      Os Disk:
        Disk Size GB:  128
        Managed Disk:
          Storage Account Type:  Premium_LRS
        Os Type:                 Linux
      Public IP:                 false
      Resource Group:            qe-jhou-hnqvw-rg
      Subnet:                    qe-jhou-hnqvw-worker-subnet
      User Data Secret:
        Name:   worker-user-data
      Vm Size:  Standard_D2s_v3_1
      Vnet:     qe-jhou-hnqvw-vnet
      Zone:     1
Status:
  Error Message:  failed to reconcile machine "qe-jhou-hnqvw-worker-centralus1-sbhpx": compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidParameter" Message="The requested VM size Standard_D2s_v3_1 is not available in the current region. The sizes available in the current region are: Standard_A0,Standard_A1,Standard_A2,Standard_A3,Standard_A5,Standard_A4,Standard_A6,Standard_A7,Basic_A0,Basic_A1,Basic_A2,Basic_A3,Basic_A4,Standard_D1_v2,Standard_D2_v2,Standard_D3_v2,Standard_D4_v2,Standard_D5_v2,Standard_D11_v2,Standard_D12_v2,Standard_D13_v2,Standard_D14_v2,Standard_D15_v2,Standard_D2_v2_Promo,Standard_D3_v2_Promo,Standard_D4_v2_Promo,Standard_D5_v2_Promo,Standard_D11_v2_Promo,Standard_D12_v2_Promo,Standard_D13_v2_Promo,Standard_D14_v2_Promo,Standard_F1,Standard_F2,Standard_F4,Standard_F8,Standard_F16,Standard_A1_v2,Standard_A2m_v2,Standard_A2_v2,Standard_A4m_v2,Standard_A4_v2,Standard_A8m_v2,Standard_A8_v2,Standard_DS1,Standard_DS2,Standard_DS3,Standard_DS4,Standard_DS11,Standard_DS12,Standard_DS13,Standard_DS14,Standard_D2_v3,Standard_D4_v3,Standard_D8_v3,Standard_D16_v3,Standard_D32_v3,Standard_D1,Standard_D2,Standard_D3,Standard_D4,Standard_D11,Standard_D12,Standard_D13,Standard_D14,Standard_F2s_v2,Standard_F4s_v2,Standard_F8s_v2,Standard_F16s_v2,Standard_F32s_v2,Standard_F48s_v2,Standard_F64s_v2,Standard_F72s_v2,Standard_B1ls,Standard_B1ms,Standard_B1s,Standard_B2ms,Standard_B2s,Standard_B4ms,Standard_B8ms,Standard_B12ms,Standard_B16ms,Standard_B20ms,Standard_DS1_v2,Standard_DS2_v2,Standard_DS3_v2,Standard_DS4_v2,Standard_DS5_v2,Standard_DS11-1_v2,Standard_DS11_v2,Standard_DS12-1_v2,Standard_DS12-2_v2,Standard_DS12_v2,Standard_DS13-2_v2,Standard_DS13-4_v2,Standard_DS13_v2,Standard_DS14-4_v2,Standard_DS14-8_v2,Standard_DS14_v2,Standard_DS2_v2_Promo,Standard_DS3_v2_Promo,Standard_DS4_v2_Promo,Standard_DS5_v2_Promo,Standard_DS11_v2_Promo,Standard_DS12_v2_Promo,Standard_DS13_v2_Promo,Standard_DS14_v2_Promo,Standard_F1s,Standard_F2s,Standard_F4s,Standard_F8s,Standard_F16s,Standard_D48_v3,Standard_D64_v3,Standard_D2s_v3,Standard_D4s_v3,Standard_D8s_v3,Standard_D16s_v3,Standard_D32s_v3,Standard_D48s_v3,Standard_D64s_v3,Standard_E2_v3,Standard_E4_v3,Standard_E8_v3,Standard_E16_v3,Standard_E20_v3,Standard_E32_v3,Standard_E48_v3,Standard_E64i_v3,Standard_E64_v3,Standard_E2s_v3,Standard_E4-2s_v3,Standard_E4s_v3,Standard_E8-2s_v3,Standard_E8-4s_v3,Standard_E8s_v3,Standard_E16-4s_v3,Standard_E16-8s_v3,Standard_E16s_v3,Standard_E20s_v3,Standard_E32-8s_v3,Standard_E32-16s_v3,Standard_E32s_v3,Standard_E48s_v3,Standard_E64-16s_v3,Standard_E64-32s_v3,Standard_E64is_v3,Standard_E64s_v3,Standard_DS15_v2,Standard_G1,Standard_G2,Standard_G3,Standard_G4,Standard_G5,Standard_GS1,Standard_GS2,Standard_GS3,Standard_GS4,Standard_GS4-4,Standard_GS4-8,Standard_GS5,Standard_GS5-8,Standard_GS5-16,Standard_L4s,Standard_L8s,Standard_L16s,Standard_L32s,Standard_HB120rs_v2.\r\nFind out more on the available VM sizes in each region at https://aka.ms/azure-regions." Target="vmSize"
  Last Updated:   2019-12-20T06:39:40Z
  Phase:          Failed


Note You need to log in before you can comment on or make changes to this bug.