Bug 2012233 - [IBMCLOUD] IPI: "Exceeded limit of remote rules per security group (the limit is 5 remote rules per security group)"
Summary: [IBMCLOUD] IPI: "Exceeded limit of remote rules per security group (the limit...
Keywords:
Status: VERIFIED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.10
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: 4.10.0
Assignee: aos-install
QA Contact: Pedro Amoedo
URL:
Whiteboard:
Depends On:
Blocks: 2009791
TreeView+ depends on / blocked
 
Reported: 2021-10-08 15:41 UTC by Christopher J Schaefer
Modified: 2021-10-13 13:51 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 5289 0 None Draft Bug 2012233: IBMCloud: Handle 5 rule SG limit 2021-10-12 00:49:51 UTC

Description Christopher J Schaefer 2021-10-08 15:41:33 UTC
Version:
4.10.0-0.nightly-2021-10-08-090421

Platform:
ibmcloud


Please specify:
IPI

What happened?

Creating a new IBM Cloud using IPI fails due to a limitation of 5 rules per SecurityGroup on IBM Cloud.

ERROR Error: Error while creating Security Group Rule Exceeded limit of remote rules per security group (the limit is 5 remote rules per security group). 
ERROR                                              
ERROR Adding a rule would exceed the limit of remote rules per security group. Consider creating another security group. 
ERROR {                                            
ERROR     "StatusCode": 400,                       
ERROR     "Headers": {                             
ERROR         "Cache-Control": [                   
ERROR             "max-age=0, no-cache, no-store, must-revalidate" 
ERROR         ],                                   
ERROR         "Cf-Cache-Status": [                 
ERROR             "DYNAMIC"                        
ERROR         ],                                   
ERROR         "Cf-Ray": [                          
ERROR             "699ffed098146653-MAD"           
ERROR         ],                                   
ERROR         "Connection": [                      
ERROR             "keep-alive"                     
ERROR         ],                                   
ERROR         "Content-Length": [                  
ERROR             "373"                            
ERROR         ],                                   
ERROR         "Content-Type": [                    
ERROR             "application/json"               
ERROR         ],                                   
ERROR         "Date": [                            
ERROR             "Wed, 06 Oct 2021 15:41:42 GMT"  
ERROR         ],                                   
ERROR         "Expect-Ct": [                       
ERROR             "max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"" 
ERROR         ],                                   
ERROR         "Expires": [                         
ERROR             "-1"                             
ERROR         ],                                   
ERROR         "Pragma": [                          
ERROR             "no-cache"                       
ERROR         ],                                   
ERROR         "Server": [                          
ERROR             "cloudflare"                     
ERROR         ],                                   
ERROR         "Strict-Transport-Security": [       
ERROR             "max-age=31536000; includeSubDomains" 
ERROR         ],                                   
ERROR         "Vary": [                            
ERROR             "Accept-Encoding"                
ERROR         ],                                   
ERROR         "X-Content-Type-Options": [          
ERROR             "nosniff"                        
ERROR         ],                                   
ERROR         "X-Request-Id": [                    
ERROR             "2f23ff2f-9acc-447a-86e4-2730be710d37" 
ERROR         ],                                   
ERROR         "X-Xss-Protection": [                
ERROR             "1; mode=block"                  
ERROR         ]                                    
ERROR     },                                       
ERROR     "Result": {                              
ERROR         "errors": [                          
ERROR             {                                
ERROR                 "code": "over_quota",        
ERROR                 "message": "Exceeded limit of remote rules per security group (the limit is 5 remote rules per security group).\n\nAdding a rule would exceed the limit of remote rules per security group. Consider creating another security group.", 
ERROR                 "more_info": "https://cloud.ibm.com/docs/vpc?topic=vpc-quotas" 
ERROR             }                                
ERROR         ],                                   
ERROR         "trace": "2f23ff2f-9acc-447a-86e4-2730be710d37" 
ERROR     },                                       
ERROR     "RawResult": null                        
ERROR }                                            
ERROR                                              
ERROR                                              
ERROR   on ../../../../tmp/openshift-install-network-003889995/vpc/security-groups.tf line 268, in resource "ibm_is_security_group_rule" "control_plane_machine_config_lb_inbound": 
ERROR  268: resource "ibm_is_security_group_rule" "control_plane_machine_config_lb_inbound" { 
ERROR                                              
ERROR                                              
FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change


What did you expect to happen?
IPI installation would properly deploy a cluster on IBM Cloud

How to reproduce it (as minimally and precisely as possible)?
Using the latest installer builds, attempt to create a new cluster on IBM Cloud
# openshift-install create cluster


Anything else we need to know?
The issue was caused by this change, putting one SG over the 5 rule limit.
https://github.com/openshift/installer/pull/5105/files#diff-b237721806432aac5fdc1b5a78484516c931af5affbbcea351e8824b5f9fcd54

IBM Cloud is attempting to find a way to add the changes above in a way that meets the 5 rule limit per SG requirement.

Comment 3 Pedro Amoedo 2021-10-13 13:51:58 UTC
[QA Summary]

[Version]

~~~
$ ./openshift-install-local version
./openshift-install-local unreleased-master-5104-g527c46172e6c4fe28838e1a18e0d758eec61fb02
built from commit 527c46172e6c4fe28838e1a18e0d758eec61fb02
release image registry.ci.openshift.org/origin/release:4.8
release architecture amd64

$ git --no-pager log --oneline --first-parent origin/master -3
527c46172 (HEAD -> master, upstream/master, origin/master, origin/HEAD) Merge pull request #5289 from cjschaef/bz_2012233
41523104b Merge pull request #5222 from santos1709/ovirt_upi_devel
681fef731 Merge pull request #5286 from rna-afk/aws_remove_tags_from_user_iam
~~~

[Environment]

~~~
apiVersion: v1
baseDomain: ibmcloud.qe.devcluster.openshift.com
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  platform:
    ibmcloud:
      type: bx2-4x16
  replicas: 2
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  platform:
    ibmcloud:
      type: bx2-8x32
  replicas: 3
metadata:
  creationTimestamp: null
  name: pamoedo-test
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineNetwork:
  - cidr: 10.0.0.0/16
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16
platform:
  ibmcloud:
    region: eu-de
    resourceGroupName: pamoedom-rg
publish: External
pullSecret:
sshKey:
~~~

[Results]

~~~
$ DIGEST=$(skopeo inspect --authfile pull-secret docker://registry.ci.openshift.org/ocp/release:4.10.0-0.nightly-2021-10-13-081040 | grep Digest | cut -d '"' -f4)
$ export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=registry.ci.openshift.org/ocp/release@${DIGEST}
$ export IC_API_KEY='<obfuscated>'
$ ./openshift-install-local create cluster --dir test37/ --log-level debug
DEBUG OpenShift Installer unreleased-master-5104-g527c46172e6c4fe28838e1a18e0d758eec61fb02 
DEBUG Built from commit 527c46172e6c4fe28838e1a18e0d758eec61fb02 
...
DEBUG Apply complete! Resources: 60 added, 0 changed, 0 destroyed. 
DEBUG                                              
DEBUG Outputs:                                     
DEBUG                                              
DEBUG control_plane_security_group_id_list = [     
DEBUG   "r010-6a6c0109-3b56-413f-8769-f2e09314654c", 
DEBUG   "r010-8624c529-7c01-4563-ac6e-7bd7d86fc470", 
DEBUG   "r010-b193d9de-0d15-4fa6-bb09-16c4f3093e20", 
DEBUG   "r010-3a5c82e4-5102-4b68-ba04-a46ce6799171", 
DEBUG ]                                            
DEBUG control_plane_subnet_id_list = [             
DEBUG   "02b7-9db6e4f0-d0d1-4fa9-b9f5-b26afb517e84", 
DEBUG   "02c7-519c5479-1142-47d8-b1bf-71848785c0d6", 
DEBUG   "02d7-1cab091f-888f-45c6-826d-20513c4eb14d", 
DEBUG ]                                            
DEBUG control_plane_subnet_zone_list = [           
DEBUG   "eu-de-1",                                 
DEBUG   "eu-de-2",                                 
DEBUG   "eu-de-3",                                 
DEBUG ]
...
~~~

*** PASSED ***


Note You need to log in before you can comment on or make changes to this bug.