Version: 4.10.0-0.nightly-2021-10-08-090421 Platform: ibmcloud Please specify: IPI What happened? Creating a new IBM Cloud using IPI fails due to a limitation of 5 rules per SecurityGroup on IBM Cloud. ERROR Error: Error while creating Security Group Rule Exceeded limit of remote rules per security group (the limit is 5 remote rules per security group). ERROR ERROR Adding a rule would exceed the limit of remote rules per security group. Consider creating another security group. ERROR { ERROR "StatusCode": 400, ERROR "Headers": { ERROR "Cache-Control": [ ERROR "max-age=0, no-cache, no-store, must-revalidate" ERROR ], ERROR "Cf-Cache-Status": [ ERROR "DYNAMIC" ERROR ], ERROR "Cf-Ray": [ ERROR "699ffed098146653-MAD" ERROR ], ERROR "Connection": [ ERROR "keep-alive" ERROR ], ERROR "Content-Length": [ ERROR "373" ERROR ], ERROR "Content-Type": [ ERROR "application/json" ERROR ], ERROR "Date": [ ERROR "Wed, 06 Oct 2021 15:41:42 GMT" ERROR ], ERROR "Expect-Ct": [ ERROR "max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"" ERROR ], ERROR "Expires": [ ERROR "-1" ERROR ], ERROR "Pragma": [ ERROR "no-cache" ERROR ], ERROR "Server": [ ERROR "cloudflare" ERROR ], ERROR "Strict-Transport-Security": [ ERROR "max-age=31536000; includeSubDomains" ERROR ], ERROR "Vary": [ ERROR "Accept-Encoding" ERROR ], ERROR "X-Content-Type-Options": [ ERROR "nosniff" ERROR ], ERROR "X-Request-Id": [ ERROR "2f23ff2f-9acc-447a-86e4-2730be710d37" ERROR ], ERROR "X-Xss-Protection": [ ERROR "1; mode=block" ERROR ] ERROR }, ERROR "Result": { ERROR "errors": [ ERROR { ERROR "code": "over_quota", ERROR "message": "Exceeded limit of remote rules per security group (the limit is 5 remote rules per security group).\n\nAdding a rule would exceed the limit of remote rules per security group. Consider creating another security group.", ERROR "more_info": "https://cloud.ibm.com/docs/vpc?topic=vpc-quotas" ERROR } ERROR ], ERROR "trace": "2f23ff2f-9acc-447a-86e4-2730be710d37" ERROR }, ERROR "RawResult": null ERROR } ERROR ERROR ERROR on ../../../../tmp/openshift-install-network-003889995/vpc/security-groups.tf line 268, in resource "ibm_is_security_group_rule" "control_plane_machine_config_lb_inbound": ERROR 268: resource "ibm_is_security_group_rule" "control_plane_machine_config_lb_inbound" { ERROR ERROR FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change What did you expect to happen? IPI installation would properly deploy a cluster on IBM Cloud How to reproduce it (as minimally and precisely as possible)? Using the latest installer builds, attempt to create a new cluster on IBM Cloud # openshift-install create cluster Anything else we need to know? The issue was caused by this change, putting one SG over the 5 rule limit. https://github.com/openshift/installer/pull/5105/files#diff-b237721806432aac5fdc1b5a78484516c931af5affbbcea351e8824b5f9fcd54 IBM Cloud is attempting to find a way to add the changes above in a way that meets the 5 rule limit per SG requirement.
[QA Summary] [Version] ~~~ $ ./openshift-install-local version ./openshift-install-local unreleased-master-5104-g527c46172e6c4fe28838e1a18e0d758eec61fb02 built from commit 527c46172e6c4fe28838e1a18e0d758eec61fb02 release image registry.ci.openshift.org/origin/release:4.8 release architecture amd64 $ git --no-pager log --oneline --first-parent origin/master -3 527c46172 (HEAD -> master, upstream/master, origin/master, origin/HEAD) Merge pull request #5289 from cjschaef/bz_2012233 41523104b Merge pull request #5222 from santos1709/ovirt_upi_devel 681fef731 Merge pull request #5286 from rna-afk/aws_remove_tags_from_user_iam ~~~ [Environment] ~~~ apiVersion: v1 baseDomain: ibmcloud.qe.devcluster.openshift.com compute: - architecture: amd64 hyperthreading: Enabled name: worker platform: ibmcloud: type: bx2-4x16 replicas: 2 controlPlane: architecture: amd64 hyperthreading: Enabled name: master platform: ibmcloud: type: bx2-8x32 replicas: 3 metadata: creationTimestamp: null name: pamoedo-test networking: clusterNetwork: - cidr: 10.128.0.0/14 hostPrefix: 23 machineNetwork: - cidr: 10.0.0.0/16 networkType: OpenShiftSDN serviceNetwork: - 172.30.0.0/16 platform: ibmcloud: region: eu-de resourceGroupName: pamoedom-rg publish: External pullSecret: sshKey: ~~~ [Results] ~~~ $ DIGEST=$(skopeo inspect --authfile pull-secret docker://registry.ci.openshift.org/ocp/release:4.10.0-0.nightly-2021-10-13-081040 | grep Digest | cut -d '"' -f4) $ export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=registry.ci.openshift.org/ocp/release@${DIGEST} $ export IC_API_KEY='<obfuscated>' $ ./openshift-install-local create cluster --dir test37/ --log-level debug DEBUG OpenShift Installer unreleased-master-5104-g527c46172e6c4fe28838e1a18e0d758eec61fb02 DEBUG Built from commit 527c46172e6c4fe28838e1a18e0d758eec61fb02 ... DEBUG Apply complete! Resources: 60 added, 0 changed, 0 destroyed. DEBUG DEBUG Outputs: DEBUG DEBUG control_plane_security_group_id_list = [ DEBUG "r010-6a6c0109-3b56-413f-8769-f2e09314654c", DEBUG "r010-8624c529-7c01-4563-ac6e-7bd7d86fc470", DEBUG "r010-b193d9de-0d15-4fa6-bb09-16c4f3093e20", DEBUG "r010-3a5c82e4-5102-4b68-ba04-a46ce6799171", DEBUG ] DEBUG control_plane_subnet_id_list = [ DEBUG "02b7-9db6e4f0-d0d1-4fa9-b9f5-b26afb517e84", DEBUG "02c7-519c5479-1142-47d8-b1bf-71848785c0d6", DEBUG "02d7-1cab091f-888f-45c6-826d-20513c4eb14d", DEBUG ] DEBUG control_plane_subnet_zone_list = [ DEBUG "eu-de-1", DEBUG "eu-de-2", DEBUG "eu-de-3", DEBUG ] ... ~~~ *** PASSED ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056