Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2087553

Summary: No rhcos-4.11/x86_64 images in the 2 new regions on alibabacloud, "ap-northeast-2 (South Korea (Seoul))" and "ap-southeast-7 (Thailand (Bangkok))"
Product: OpenShift Container Platform Reporter: Jianli Wei <jiwei>
Component: RHCOSAssignee: Michael Nguyen <mnguyen>
Status: CLOSED ERRATA QA Contact: Jianli Wei <jiwei>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.11CC: dornelas, dustymabe, gpei, jligon, miabbott, mnguyen, mrussell, nstielau, travier
Target Milestone: ---   
Target Release: 4.13.0   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Cause: The 2 regions are newly supported by Alibabacloud. Consequence: ,The RHCOS images are not yet available in these regions. Workaround (if any): n/a Result: OCP installation would fail in these regions.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2023-05-17 22:46:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jianli Wei 2022-05-18 03:22:08 UTC 
OCP Version at Install Time: registry.ci.openshift.org/ocp/release:4.11.0-0.nightly-2022-05-11-054135
Platform: alibabacloud
Architecture: x86_64


What are you trying to do? What is your use case?
IPI install an OCP cluster on alibabacloud in these new regions.


What happened? What went wrong or what did you expect?
No rhcos-4.11/x86_64 images in the 2 new regions on alibabacloud, i.e. "ap-northeast-2 (South Korea (Seoul))" and "ap-southeast-7 (Thailand (Bangkok))", so that OCP IPI installation would fail.


What are the steps to reproduce your issue? Please try to reduce these steps to something that can be reproduced with a single RHCOS node.
Always, when trying to IPI install OCP cluster in these regions on alibabacloud.


If you're having problems booting/installing RHCOS, please provide:
>the 2 new regions, "ap-northeast-2 (South Korea (Seoul))" and "ap-southeast-7 (Thailand (Bangkok))", are not present in below command output:
$ openshift-install version
openshift-install 4.11.0-0.nightly-2022-05-11-054135
built from commit 248eeca22322c2ad1634ae6b8d54c7637e3ff4a9
release image registry.ci.openshift.org/ocp/release@sha256:5967359c2bfee0512030418af0f69faa3fa74a81a89ad64a734420e020e7f107
release architecture amd64
$ openshift-install coreos print-stream-json | jq -r '.architectures.x86_64.images.aliyun'
{
  "regions": {
    "ap-northeast-1": {
      "release": "411.85.202203181601-0",
      "image": "m-6weev9pf64cymu161jhc"
    },
    "ap-south-1": {
      "release": "411.85.202203181601-0",
      "image": "m-a2d9i7idzzzd3rag1lf0"
    },
    "ap-southeast-1": {
      "release": "411.85.202203181601-0",
      "image": "m-t4n168vzz5f16ms5kqbr"
    },
    "ap-southeast-2": {
      "release": "411.85.202203181601-0",
      "image": "m-p0wexfrw72q08v5j6e4e"
    },
    "ap-southeast-3": {
      "release": "411.85.202203181601-0",
      "image": "m-8ps3qahwrxdh7x702osl"
    },
    "ap-southeast-5": {
      "release": "411.85.202203181601-0",
      "image": "m-k1ae6rawoqtkxvuz7myd"
    },
    "ap-southeast-6": {
      "release": "411.85.202203181601-0",
      "image": "m-5tsim6n7d4xot9i74avt"
    },
    "cn-beijing": {
      "release": "411.85.202203181601-0",
      "image": "m-2zedvnyg9y0p30tilebt"
    },
    "cn-chengdu": {
      "release": "411.85.202203181601-0",
      "image": "m-2vc97qcawcm72ejwevwx"
    },
    "cn-guangzhou": {
      "release": "411.85.202203181601-0",
      "image": "m-7xv787bto66ikaqeqgi8"
    },
    "cn-hangzhou": {
      "release": "411.85.202203181601-0",
      "image": "m-bp1243k7dew81hcws7ge"
    },
    "cn-heyuan": {
      "release": "411.85.202203181601-0",
      "image": "m-f8zin3xo0wr0378mnkxu"
    },
    "cn-hongkong": {
      "release": "411.85.202203181601-0",
      "image": "m-j6c9a603yxnmrcr0k0xe"
    },
    "cn-huhehaote": {
      "release": "411.85.202203181601-0",
      "image": "m-hp3csos7jfan9rc2umgv"
    },
    "cn-nanjing": {
      "release": "411.85.202203181601-0",
      "image": "m-gc7dk6o21jwbxh8t0iol"
    },
    "cn-qingdao": {
      "release": "411.85.202203181601-0",
      "image": "m-m5eb8cxeiib19sa1u3a0"
    },
    "cn-shanghai": {
      "release": "411.85.202203181601-0",
      "image": "m-uf66cz852fh3y7rx0iph"
    },
    "cn-shenzhen": {
      "release": "411.85.202203181601-0",
      "image": "m-wz98ej4pmj4hbmjvoomk"
    },
    "cn-wulanchabu": {
      "release": "411.85.202203181601-0",
      "image": "m-0jl1j22cbl19aktv0tjw"
    },
    "cn-zhangjiakou": {
      "release": "411.85.202203181601-0",
      "image": "m-8vbiwk22jkjualnxlpp7"
    },
    "eu-central-1": {
      "release": "411.85.202203181601-0",
      "image": "m-gw8ie7hjr9b6ljzylkvk"
    },
    "eu-west-1": {
      "release": "411.85.202203181601-0",
      "image": "m-d7o7ousvo8ekohx5rfxw"
    },
    "me-east-1": {
      "release": "411.85.202203181601-0",
      "image": "m-eb3bs50k4hihjjjeoj04"
    },
    "us-east-1": {
      "release": "411.85.202203181601-0",
      "image": "m-0xi29kf08acv9dps47zs"
    },
    "us-west-1": {
      "release": "411.85.202203181601-0",
      "image": "m-rj92cd5lh2mbft6v6bfv"
    }
  }
}
$ 


Please add anything else that might be useful, for example:
>OCP interactive installer tells the 2 new regions:
$ openshift-install create install-config --dir work
? SSH Public Key /home/fedora/.ssh/openshift-qe.pub
? Platform alibabacloud
? Region  [Use arrows to move, type to filter, ? for more help]
  ap-northeast-1 (Japan (Tokyo))
> ap-northeast-2 (South Korea (Seoul))
  ap-south-1 (India (Mumbai))
  ap-southeast-1 (Singapore)
  ap-southeast-2 (Australia (Sydney))
  ap-southeast-3 (Malaysia (Kuala Lumpur))
  ap-southeast-5 (Indonesia (Jakarta))
  ap-southeast-6 (Philippines (Manila))
  ap-southeast-7 (Thailand (Bangkok))
...


>Trying to create manifests with valid install-config.yaml got errors of "No Aliyun images in region ap-northeast-2" / "No Aliyun images in region ap-southeast-7".
$ yq r work/install-config.yaml platform
alibabacloud:
  region: ap-northeast-2
$ yq r work/install-config.yaml compute
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  platform:
    alibabacloud:
      instanceType: ecs.g6e.large
      systemDiskSize: 128
  replicas: 2
$ yq r work/install-config.yaml controlPlane
architecture: amd64
hyperthreading: Enabled
name: master
platform:
  alibabacloud:
    instanceType: ecs.g6e.xlarge
    systemDiskSize: 128
replicas: 3
$ yq r work/install-config.yaml credentialsMode
Manual
$ openshift-install create manifests --dir work
FATAL failed to fetch Master Machines: failed to fetch dependency of "Master Machines": failed to generate asset "Image": rhcos-4.11/x86_64: No Aliyun images in region ap-northeast-2
$ 
$ 
$ yq r work2/install-config.yaml platform
alibabacloud:
  region: ap-southeast-7
$ yq r work2/install-config.yaml compute
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  platform:
    alibabacloud:
      instanceType: ecs.g6e.large
      systemDiskSize: 128
  replicas: 2
$ yq r work2/install-config.yaml controlPlane
architecture: amd64
hyperthreading: Enabled
name: master
platform:
  alibabacloud:
    instanceType: ecs.g6e.xlarge
    systemDiskSize: 128
replicas: 3
$ yq r work2/install-config.yaml credentialsMode
Manual
$ openshift-install create manifests --dir work2
FATAL failed to fetch Master Machines: failed to fetch dependency of "Master Machines": failed to generate asset "Image": rhcos-4.11/x86_64: No Aliyun images in region ap-southeast-7 
$
Comment 4 Micah Abbott 2022-06-03 14:02:59 UTC 
After discussing with PM, we are going to target this for 4.12 - https://issues.redhat.com/browse/COS-1571

There is no Target Release for 4.12 yet, so this will sit untargeted for a while
Comment 5 Dusty Mabe 2022-12-12 04:11:32 UTC 
We recently switched to a new pipeline for building RHCOS and these regions now get uploaded to. IIUC the next releases (on all streams) shipped to customers should have these regions available.
Comment 6 Timothée Ravier 2022-12-12 09:34:05 UTC 
Re-targeting this one for 4.13. If we want to have notes for all releases we can clone it but not sure it's worth it.
Comment 8 Jianli Wei 2022-12-28 02:55:00 UTC 
Verified with the latest nightly builds of 4.11, 4.12 and 4.13, see below.

[fedora@preserve-jiwei ~]$ cd ocp411
[fedora@preserve-jiwei ocp411]$ ./openshift-install version
./openshift-install 4.11.0-0.nightly-2022-12-26-210225
built from commit 616d13577a49a887c7adc0236e22903dfcdb1ac5
release image registry.ci.openshift.org/ocp/release@sha256:cf4b97844498dfb1be3180c474212d872a713e00b24208ed5ed83ff45fce7506
release architecture amd64
[fedora@preserve-jiwei ocp411]$ ./openshift-install coreos print-stream-json | jq -r ".architectures.x86_64.images.aliyun"
{
  "regions": {
    "ap-northeast-1": {
      "release": "411.86.202212072103-0",
      "image": "m-6we84mbluxc5oc7zyv3w"
    },
    "ap-northeast-2": {
      "release": "411.86.202212072103-0",
      "image": "m-mj7dhe4tzdhxqz7y327w"
    },
    "ap-south-1": {
      "release": "411.86.202212072103-0",
      "image": "m-a2d7hgpmhomwjc2ro6eb"
    },
    "ap-southeast-1": {
      "release": "411.86.202212072103-0",
      "image": "m-t4nfkfl0za0s38dmyzui"
    },
    "ap-southeast-2": {
      "release": "411.86.202212072103-0",
      "image": "m-p0web0c4yfr0m3bgh93t"
    },
    "ap-southeast-3": {
      "release": "411.86.202212072103-0",
      "image": "m-8ps8wr2mes2ymqswczjq"
    },
    "ap-southeast-5": {
      "release": "411.86.202212072103-0",
      "image": "m-k1a59t4efwq6qlxwah7f"
    },
    "ap-southeast-6": {
      "release": "411.86.202212072103-0",
      "image": "m-5tsa54liiffu1ilbkqzs"
    },
    "ap-southeast-7": {
      "release": "411.86.202212072103-0",
      "image": "m-0jod1und9ze6vt46w8og"
    },
    "cn-beijing": {
      "release": "411.86.202212072103-0",
      "image": "m-2ze0x6lkpz211t7o4trs"
    },
    "cn-chengdu": {
      "release": "411.86.202212072103-0",
      "image": "m-2vc583dmdrys5ghrab1v"
    },
    "cn-fuzhou": {
      "release": "411.86.202212072103-0",
      "image": "m-gw02ocyshbcsy4nkiu3s"
    },
    "cn-guangzhou": {
      "release": "411.86.202212072103-0",
      "image": "m-7xv6yupw3qlo1aw55lzd"
    },
    "cn-hangzhou": {
      "release": "411.86.202212072103-0",
      "image": "m-bp1htdxchchh9rr9gtqe"
    },
    "cn-heyuan": {
      "release": "411.86.202212072103-0",
      "image": "m-f8zi3fosukjbq8mm18j2"
    },
    "cn-hongkong": {
      "release": "411.86.202212072103-0",
      "image": "m-j6c7z01zqicjiqrjl859"
    },
    "cn-huhehaote": {
      "release": "411.86.202212072103-0",
      "image": "m-hp30oy01z49fc0iq6zg0"
    },
    "cn-nanjing": {
      "release": "411.86.202212072103-0",
      "image": "m-gc7ggmvfpx2pym440g9a"
    },
    "cn-qingdao": {
      "release": "411.86.202212072103-0",
      "image": "m-m5ea1becv9sdofl71mrr"
    },
    "cn-shanghai": {
      "release": "411.86.202212072103-0",
      "image": "m-uf653z2czwo7tbd82tch"
    },
    "cn-shenzhen": {
      "release": "411.86.202212072103-0",
      "image": "m-wz9b4eo4xsezdr780azs"
    },
    "cn-wulanchabu": {
      "release": "411.86.202212072103-0",
      "image": "m-0jlbzivalq5miliazsqk"
    },
    "cn-zhangjiakou": {
      "release": "411.86.202212072103-0",
      "image": "m-8vb9ztjs977e6g9fqc6o"
    },
    "eu-central-1": {
      "release": "411.86.202212072103-0",
      "image": "m-gw8blhu5tzokwpgyp97u"
    },
    "eu-west-1": {
      "release": "411.86.202212072103-0",
      "image": "m-d7o632ny4rywe6n1rww8"
    },
    "me-east-1": {
      "release": "411.86.202212072103-0",
      "image": "m-eb3hfys9xoel41dswj2h"
    },
    "us-east-1": {
      "release": "411.86.202212072103-0",
      "image": "m-0xif9y7r0fst7lihbfxx"
    },
    "us-west-1": {
      "release": "411.86.202212072103-0",
      "image": "m-rj9gjdatfp30mdlpe2fn"
    }
  }
}
[fedora@preserve-jiwei ocp411]$ 
[fedora@preserve-jiwei ocp411]$ cd ../ocp412
[fedora@preserve-jiwei ocp412]$ ./openshift-install version
./openshift-install 4.12.0-0.nightly-2022-12-27-111646
built from commit 9e4a327874648ae9ec89a4d23d85e202d543d6e8
release image registry.ci.openshift.org/ocp/release@sha256:576d3e6832dce19e4dde7dd9b7664847f71dbbf322b2a9a35384346b8d39a6cd
release architecture amd64
[fedora@preserve-jiwei ocp412]$ ./openshift-install coreos print-stream-json | jq -r ".architectures.x86_64.images.aliyun"
{
  "regions": {
    "ap-northeast-1": {
      "release": "412.86.202212081411-0",
      "image": "m-6we92hy9gyvrdwktiiaz"
    },
    "ap-northeast-2": {
      "release": "412.86.202212081411-0",
      "image": "m-mj7djef4a1g5vi1m66ry"
    },
    "ap-south-1": {
      "release": "412.86.202212081411-0",
      "image": "m-a2d7hgpmhomvvlqdwf2u"
    },
    "ap-southeast-1": {
      "release": "412.86.202212081411-0",
      "image": "m-t4n99kheklsvyuw79bcl"
    },
    "ap-southeast-2": {
      "release": "412.86.202212081411-0",
      "image": "m-p0wiz3t1y6eo4qmliiqe"
    },
    "ap-southeast-3": {
      "release": "412.86.202212081411-0",
      "image": "m-8psbyqmzqmloqi8q4mgx"
    },
    "ap-southeast-5": {
      "release": "412.86.202212081411-0",
      "image": "m-k1aecz2x6o6ht7w0bqdf"
    },
    "ap-southeast-6": {
      "release": "412.86.202212081411-0",
      "image": "m-5tsj1yxloi3prfq3jern"
    },
    "ap-southeast-7": {
      "release": "412.86.202212081411-0",
      "image": "m-0jod1und9ze66rf2f5yi"
    },
    "cn-beijing": {
      "release": "412.86.202212081411-0",
      "image": "m-2zedxhk98442fgn5hg7o"
    },
    "cn-chengdu": {
      "release": "412.86.202212081411-0",
      "image": "m-2vc3k0f06j9j3y9o4ft8"
    },
    "cn-fuzhou": {
      "release": "412.86.202212081411-0",
      "image": "m-gw0edjtnt2ztra5v4mfg"
    },
    "cn-guangzhou": {
      "release": "412.86.202212081411-0",
      "image": "m-7xvg5o6cwctseoufn7o1"
    },
    "cn-hangzhou": {
      "release": "412.86.202212081411-0",
      "image": "m-bp10iiqoivwiqky0fw9m"
    },
    "cn-heyuan": {
      "release": "412.86.202212081411-0",
      "image": "m-f8zbicl9f82nnn3uf0lu"
    },
    "cn-hongkong": {
      "release": "412.86.202212081411-0",
      "image": "m-j6cj81shutf7xs9ci8j0"
    },
    "cn-huhehaote": {
      "release": "412.86.202212081411-0",
      "image": "m-hp3g86cstpjtz8r6pzks"
    },
    "cn-nanjing": {
      "release": "412.86.202212081411-0",
      "image": "m-gc75rbpctjy2ona8tt1b"
    },
    "cn-qingdao": {
      "release": "412.86.202212081411-0",
      "image": "m-m5e9felqzdvggvp5016o"
    },
    "cn-shanghai": {
      "release": "412.86.202212081411-0",
      "image": "m-uf6c04i5ofxt1rc7qzdl"
    },
    "cn-shenzhen": {
      "release": "412.86.202212081411-0",
      "image": "m-wz97mva2hlrd8b31o3lc"
    },
    "cn-wulanchabu": {
      "release": "412.86.202212081411-0",
      "image": "m-0jle2acx8y8rgutbk923"
    },
    "cn-zhangjiakou": {
      "release": "412.86.202212081411-0",
      "image": "m-8vbcskxc4svqp6m32she"
    },
    "eu-central-1": {
      "release": "412.86.202212081411-0",
      "image": "m-gw86sffy6f512xceq0sn"
    },
    "eu-west-1": {
      "release": "412.86.202212081411-0",
      "image": "m-d7oe2alj2ootweztynbl"
    },
    "me-east-1": {
      "release": "412.86.202212081411-0",
      "image": "m-eb32k3q3yp9vnql6vx8h"
    },
    "us-east-1": {
      "release": "412.86.202212081411-0",
      "image": "m-0xi67ek28y49owje0n45"
    },
    "us-west-1": {
      "release": "412.86.202212081411-0",
      "image": "m-rj9cuag5oy9pvbugddm7"
    }
  }
}
[fedora@preserve-jiwei ocp412]$ 
[fedora@preserve-jiwei ocp412]$ cd ../ocp413
[fedora@preserve-jiwei ocp413]$ ./openshift-install version
./openshift-install 4.13.0-0.nightly-2022-12-23-223710
built from commit 52b495c654ebf90dc96bfb162a156f535218bbfc
release image registry.ci.openshift.org/ocp/release@sha256:76259072b4df3c6abbfc66f84795b94cd7bd632198783e4a24f8295ffd928f23
release architecture amd64
[fedora@preserve-jiwei ocp413]$ ./openshift-install coreos print-stream-json | jq -r ".architectures.x86_64.images.aliyun"
{
  "regions": {
    "ap-northeast-1": {
      "release": "413.86.202212131234-0",
      "image": "m-6we6feq2a5g7p5frcyye"
    },
    "ap-northeast-2": {
      "release": "413.86.202212131234-0",
      "image": "m-mj7c3au3ndsfcd73qi7g"
    },
    "ap-south-1": {
      "release": "413.86.202212131234-0",
      "image": "m-a2d9hlz9nkguhcka84ce"
    },
    "ap-southeast-1": {
      "release": "413.86.202212131234-0",
      "image": "m-t4ndtsvulkrd473qv0qw"
    },
    "ap-southeast-2": {
      "release": "413.86.202212131234-0",
      "image": "m-p0w0xqu07c3mc9hhovjj"
    },
    "ap-southeast-3": {
      "release": "413.86.202212131234-0",
      "image": "m-8ps5df36ty01e9z12fek"
    },
    "ap-southeast-5": {
      "release": "413.86.202212131234-0",
      "image": "m-k1aecz2x6o6hszzvvigy"
    },
    "ap-southeast-6": {
      "release": "413.86.202212131234-0",
      "image": "m-5tsj1yxloi3prfq3jerm"
    },
    "ap-southeast-7": {
      "release": "413.86.202212131234-0",
      "image": "m-0joevsjjsbd0yrvzeniv"
    },
    "cn-beijing": {
      "release": "413.86.202212131234-0",
      "image": "m-2zeb7nk0s4ept6wpngow"
    },
    "cn-chengdu": {
      "release": "413.86.202212131234-0",
      "image": "m-2vcdvpqg0c17dllqlucb"
    },
    "cn-fuzhou": {
      "release": "413.86.202212131234-0",
      "image": "m-gw04uo3jlikgnuailw67"
    },
    "cn-guangzhou": {
      "release": "413.86.202212131234-0",
      "image": "m-7xvg5o6cwctse735mpw0"
    },
    "cn-hangzhou": {
      "release": "413.86.202212131234-0",
      "image": "m-bp1ftlbdilr2q1y1zyb4"
    },
    "cn-heyuan": {
      "release": "413.86.202212131234-0",
      "image": "m-f8zb4p794i7lho3k3i1m"
    },
    "cn-hongkong": {
      "release": "413.86.202212131234-0",
      "image": "m-j6cj81shutf7xs9ci8j1"
    },
    "cn-huhehaote": {
      "release": "413.86.202212131234-0",
      "image": "m-hp3gg9slg95gexd0586t"
    },
    "cn-nanjing": {
      "release": "413.86.202212131234-0",
      "image": "m-gc7edjtnt2ztr86u0kgb"
    },
    "cn-qingdao": {
      "release": "413.86.202212131234-0",
      "image": "m-m5e7wz04axz3pzoinf9e"
    },
    "cn-shanghai": {
      "release": "413.86.202212131234-0",
      "image": "m-uf6fw94i35t7vls5mh4i"
    },
    "cn-shenzhen": {
      "release": "413.86.202212131234-0",
      "image": "m-wz9d5x3k9pn1d96d9vv7"
    },
    "cn-wulanchabu": {
      "release": "413.86.202212131234-0",
      "image": "m-0jl91t3p5kehj1ffeoqf"
    },
    "cn-zhangjiakou": {
      "release": "413.86.202212131234-0",
      "image": "m-8vb9ztjs977di09nhv60"
    },
    "eu-central-1": {
      "release": "413.86.202212131234-0",
      "image": "m-gw8jfcr55yhzzct8dpdx"
    },
    "eu-west-1": {
      "release": "413.86.202212131234-0",
      "image": "m-d7obif53ntpwvjl9sq96"
    },
    "me-east-1": {
      "release": "413.86.202212131234-0",
      "image": "m-eb33ede6siwgblm7zbzi"
    },
    "us-east-1": {
      "release": "413.86.202212131234-0",
      "image": "m-0xi02hsc7wmc839n1xr5"
    },
    "us-west-1": {
      "release": "413.86.202212131234-0",
      "image": "m-rj95xgeh5qxxgaiykxvw"
    }
  }
}
[fedora@preserve-jiwei ocp413]$
Comment 11 errata-xmlrpc 2023-05-17 22:46:32 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.13.0 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:1326