Bug 1899853

Summary: additionalSecurityGroupIDs not working for master nodes
Product: OpenShift Container Platform Reporter: Robert Heinzmann <rheinzma>
Component: InstallerAssignee: Pierre Prinetti <pprinett>
Installer sub component: OpenShift on OpenStack QA Contact: weiwei jiang <wjiang>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: maemmanu, mleonard, pprinett
Version: 4.6Keywords: UpcomingSprint
Target Milestone: ---   
Target Release: 4.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Control Plane ports were not assigned the additional user-defined Security groups Consequence: Additional user-defined Security group rules were not properly applied to Control plane nodes Fix: The additional user-defined Security groups are now assigned to the Control plane nodes ports Result: Additional user-defined security groups now correctly apply to Control plane nodes.
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-02-24 15:35:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1901736    

Description Robert Heinzmann 2020-11-20 09:01:26 UTC
Version:

$ openshift-install version

[stack@osp16 test-additional-sg]$ ../openshift-install version
../openshift-install 4.6.3
built from commit a4f0869e0d2a5b2d645f0f28ef9e4b100fa8f779
release image 192.168.100.98:443/ocp4/openshift4@sha256:14986d2b9c112ca955aaa03f7157beadda0bd3c089e5e1d56f28020d2dd55c52

Platform:

OpenStack 16.1

Please specify:

IPI (automated install with `openshift-install`. If you don't know, then it's IPI)


What happened?

Configuring additionalSecurityGroupIDs for the masters does not work. Cluster is bootstrapped, however additional security groups are missing. 

~~~
controlPlane:
  hyperthreading: Enabled
  architecture: amd64
  name: master
  platform:
    openstack:
      type: openshift.master
      additionalSecurityGroupIDs:
      - 2959554a-8cca-4260-82bf-e0fcbb87f40c
  replicas: 3
~~~

The additional security group can not be found on the resulting servers

~~~
[stack@osp16 ocp-test1]$ openstack server show ocp-99l7h-master-0
+-----------------------------+----------------------------------------------------------+
| Field                       | Value                                                    |
+-----------------------------+----------------------------------------------------------+
| id                          | d6c5466a-b43a-47c2-84a4-41afe574d3f2                     |
| name                        | ocp-99l7h-master-0                                       |
| security_groups             | name='ocp-99l7h-master'                                  
~~~

I would suspect that https://github.com/openshift/installer/blob/release-4.6/data/data/openstack/topology/private-network.tf#L46 is missing the additional ID's that are configured here https://github.com/openshift/installer/blob/release-4.6/data/data/openstack/masters/main.tf#L49

Reproducer: http://pastebin.test.redhat.com/919822
Fixed Reproducer: http://pastebin.test.redhat.com/919824

It seems that port security group settings overrule machine settings.

What did you expect to happen?

Masters get additional security groups

How to reproduce it (as minimally and precisely as possible)?

Create additional security group
Configure additionalSecurityGroupIDs in the installconfig
Install IPI
verify the security groups

Comment 1 Pierre Prinetti 2020-11-25 11:24:29 UTC
A question from Martin on Github[1]:

> The patch fixes exactly the issue that is reported in the BZ so I do not see any reason not to approve.
> It would be nice to know if we also need to apply the additional security groups to the VIP ports. Otherwise
> what is the point of adding security groups at all to api_port and ingress_port?


@robert: do we cover the use case by just setting the additional SGs on the master ports, or should we go beyond that to deliver value for the customer?


[1]: https://github.com/openshift/installer/pull/4411#pullrequestreview-538216817

Comment 2 Maria Emmanuelli 2020-11-25 13:14:01 UTC
@pierre The SG should be set only to the master ports and not to the VIP ports. It should follow the same behaviour as with the worker nodes.

Comment 3 Pierre Prinetti 2020-11-25 13:27:45 UTC
Perfect, thanks.

Comment 5 weiwei jiang 2020-11-26 07:12:39 UTC
Checked with latest
./openshift-install 4.7.0-0.nightly-2020-11-25-114114
built from commit a9e6c4d8fa0e7d5edb9cf95330689a65261ff09c
release image registry.svc.ci.openshift.org/ocp/release@sha256:bf37e13af0e254d0b744b62ace0dcf5560230374d7877a8fde16cf9134ec7862

---
apiVersion: v1
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  platform:
    openstack:
      additionalSecurityGroupIDs: &1
      - 8794f45c-4f54-40a4-aadb-38d6c32e286e
  replicas: 3
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  platform:
    openstack:
      additionalSecurityGroupIDs: *1
      type: m1.large
  replicas: 3
metadata:
  name: wj47ios1126y
platform:
  openstack:
    cloud: openstack
    computeFlavor: m1.xlarge
    region: regionOne
    trunkSupport: '1'
    octaviaSupport: '0'
    lbFloatingIP: 10.0.102.125
    ingressFloatingIP: 10.0.103.227
    externalNetwork: provider_net_cci_8
pullSecret: HIDDEN
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  serviceNetwork:
  - 172.30.0.0/16
  machineNetwork:
  - cidr: 192.168.0.0/18
  networkType: OpenShiftSDN
publish: External
baseDomain: 1126-7gp.qe.rhcloud.com
sshKey: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCWkwurd8TNAi+D7ffvyDdhGBSQtJx3/Yedlwvvha0q772vLlOAGlKCw4dajKy6qty1/GGQDgTJ17h3C9TEArI8ZqILnyydeY56DL+ELN3dtGBVof/N2qtW0+SmEnd1Mi7Qy5Tx4e/GVmB3NgX9szwNOVXhebzgBsXc9x+RtCVLPLC8J+qqSdTUZ0UfJsh2ptlQLGHmmTpF//QlJ1tngvAFeCOxJUhrLAa37P9MtFsiNk31EfKyBk3eIdZljTERmqFaoJCohsFFEdO7tVgU6p5NwniAyBGZVjZBzjELoI1aZ+/g9yReIScxl1R6PWqEzcU6lGo2hInnb6nuZFGb+90D
  openshift-qe

and the control-plane still not take affect, need wait new payload to have a try.

# openstack server show wj47ios1126y-xb8px-master-2
+-----------------------------+---------------------------------------------------------------------------+
| Field                       | Value                                                                     |
+-----------------------------+---------------------------------------------------------------------------+
| OS-DCF:diskConfig           | MANUAL                                                                    |
| OS-EXT-AZ:availability_zone | nova                                                                      |
| OS-EXT-STS:power_state      | Running                                                                   |
| OS-EXT-STS:task_state       | None                                                                      |
| OS-EXT-STS:vm_state         | active                                                                    |
| OS-SRV-USG:launched_at      | 2020-11-26T06:24:13.000000                                                |
| OS-SRV-USG:terminated_at    | None                                                                      |
| accessIPv4                  |                                                                           |
| accessIPv6                  |                                                                           |
| addresses                   | wj47ios1126y-xb8px-openshift=192.168.2.137                                |
| config_drive                |                                                                           |
| created                     | 2020-11-26T06:23:21Z                                                      |
| flavor                      | m1.xlarge (3f183920-6cba-4bfb-ab3a-599559cf0f97)                          |
| hostId                      | fa75360746cc026112b3802a27c06948313b684d2f736724e3482ad4                  |
| id                          | 4ed158a9-cd4f-440a-85ba-5c55cf0a40d9                                      |
| image                       | wj47ios1126y-xb8px-rhcos (64a046b8-0f3b-44d9-b51d-83dcbe9cb928)           |
| key_name                    | None                                                                      |
| name                        | wj47ios1126y-xb8px-master-2                                               |
| progress                    | 0                                                                         |
| project_id                  | 542c6ebd48bf40fa857fc245c7572e30                                          |
| properties                  | Name='wj47ios1126y-xb8px-master', openshiftClusterID='wj47ios1126y-xb8px' |
| security_groups             | name='wj47ios1126y-xb8px-master'                                          |
| status                      | ACTIVE                                                                    |
| updated                     | 2020-11-26T06:24:13Z                                                      |
| user_id                     | b414646065ab99780ef1bbcba52c07d2033a6f99fd0b10a3b1b12fcb5e5275e1          |
| volumes_attached            |                                                                           |
+-----------------------------+---------------------------------------------------------------------------+
[root@wjiang-bind-bastion ~]# openstack server show wj47ios1126y-xb8px-worker-0-dmvtk
+-----------------------------+---------------------------------------------------------------------------+
| Field                       | Value                                                                     |
+-----------------------------+---------------------------------------------------------------------------+
| OS-DCF:diskConfig           | MANUAL                                                                    |
| OS-EXT-AZ:availability_zone | nova                                                                      |
| OS-EXT-STS:power_state      | Running                                                                   |
| OS-EXT-STS:task_state       | None                                                                      |
| OS-EXT-STS:vm_state         | active                                                                    |
| OS-SRV-USG:launched_at      | 2020-11-26T06:50:50.000000                                                |
| OS-SRV-USG:terminated_at    | None                                                                      |
| accessIPv4                  |                                                                           |
| accessIPv6                  |                                                                           |
| addresses                   | wj47ios1126y-xb8px-openshift=192.168.3.154                                |
| config_drive                |                                                                           |
| created                     | 2020-11-26T06:50:26Z                                                      |
| flavor                      | m1.large (a9acc2de-39d7-4148-8d16-413c3b696e9d)                           |
| hostId                      | fa75360746cc026112b3802a27c06948313b684d2f736724e3482ad4                  |
| id                          | c5949b79-a8a9-4cd0-a0e0-6fe37f567270                                      |
| image                       | wj47ios1126y-xb8px-rhcos (64a046b8-0f3b-44d9-b51d-83dcbe9cb928)           |
| key_name                    | None                                                                      |
| name                        | wj47ios1126y-xb8px-worker-0-dmvtk                                         |
| progress                    | 0                                                                         |
| project_id                  | 542c6ebd48bf40fa857fc245c7572e30                                          |
| properties                  | Name='wj47ios1126y-xb8px-worker', openshiftClusterID='wj47ios1126y-xb8px' |
| security_groups             | name='wj47ios1126y-xb8px-worker'                                          |
|                             | name='default'                                                            |
| status                      | ACTIVE                                                                    |
| updated                     | 2020-11-26T06:50:50Z                                                      |
| user_id                     | b414646065ab99780ef1bbcba52c07d2033a6f99fd0b10a3b1b12fcb5e5275e1          |
| volumes_attached            |                                                                           |
+-----------------------------+---------------------------------------------------------------------------+

Comment 6 weiwei jiang 2020-11-27 02:22:46 UTC
Checked with latest payload
./openshift-install 4.7.0-0.nightly-2020-11-26-221840
built from commit 64ec239bc596635b50dd82485c9932cdf10c861e
release image registry.svc.ci.openshift.org/ocp/release@sha256:542e9447623e5e5f0ba96be505d695b81b7b0b088452a19d66b0c4f1e0f6654b

with install-config.yaml:
---
apiVersion: v1
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  platform:
    openstack:
      additionalSecurityGroupIDs: &1
      - 8794f45c-4f54-40a4-aadb-38d6c32e286e
  replicas: 3
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  platform:
    openstack:
      additionalSecurityGroupIDs: *1
      type: m1.large
  replicas: 3
metadata:
  name: wj47ios1127z
platform:
  openstack:
    cloud: openstack
    computeFlavor: m1.xlarge
    region: regionOne
    trunkSupport: '1'
    octaviaSupport: '0'
    lbFloatingIP: 10.0.103.36
    ingressFloatingIP: 10.0.103.31
    externalNetwork: provider_net_cci_8
pullSecret: HIDDEN
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  serviceNetwork:
  - 172.30.0.0/16
  machineNetwork:
  - cidr: 192.168.0.0/18
  networkType: OpenShiftSDN
publish: External
baseDomain: 1127-ggc.qe.rhcloud.com
sshKey: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCWkwurd8TNAi+D7ffvyDdhGBSQtJx3/Yedlwvvha0q772vLlOAGlKCw4dajKy6qty1/GGQDgTJ17h3C9TEArI8ZqILnyydeY56DL+ELN3dtGBVof/N2qtW0+SmEnd1Mi7Qy5Tx4e/GVmB3NgX9szwNOVXhebzgBsXc9x+RtCVLPLC8J+qqSdTUZ0UfJsh2ptlQLGHmmTpF//QlJ1tngvAFeCOxJUhrLAa37P9MtFsiNk31EfKyBk3eIdZljTERmqFaoJCohsFFEdO7tVgU6p5NwniAyBGZVjZBzjELoI1aZ+/g9yReIScxl1R6PWqEzcU6lGo2hInnb6nuZFGb+90D
  openshift-qe


# openstack server list  --name wj47ios1127z-9mdcj -f value -c ID | xargs -n 1 openstack server show -f json|jq -r '"===========\n"+"Server: "+ .name, "Security Groups:\n" + .security_groups'                                                    
===========
Server: wj47ios1127z-9mdcj-worker-0-l8n4w
Security Groups:
name='wj47ios1127z-9mdcj-worker'
name='default'
===========
Server: wj47ios1127z-9mdcj-worker-0-kb2wj
Security Groups:
name='wj47ios1127z-9mdcj-worker'
name='default'
===========
Server: wj47ios1127z-9mdcj-worker-0-dx2v9
Security Groups:
name='wj47ios1127z-9mdcj-worker'
name='default'
===========
Server: wj47ios1127z-9mdcj-master-2
Security Groups:
name='default'
name='wj47ios1127z-9mdcj-master'
===========
Server: wj47ios1127z-9mdcj-master-1
Security Groups:
name='default'
name='wj47ios1127z-9mdcj-master'
===========
Server: wj47ios1127z-9mdcj-master-0
Security Groups:
name='default'
name='wj47ios1127z-9mdcj-master'

Comment 8 milti leonard 2020-12-07 18:23:43 UTC
is there anything further that needs to be done on this? also, can this be back-ported to 4.5/4.6?

thnx, m

Comment 9 Pierre Prinetti 2020-12-07 20:08:42 UTC
(In reply to milti leonard from comment #8)
> is there anything further that needs to be done on this? also, can this be
> back-ported to 4.5/4.6?
> 
> thnx, m

The 4.6 patch for is waiting for the release-manager's approval https://github.com/openshift/installer/pull/4420

Comment 14 errata-xmlrpc 2021-02-24 15:35:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633