Bug 1943219

Summary: unable to install IPI PRIVATE OpenShift cluster in Azure - SSH access from the Internet should be blocked
Product: OpenShift Container Platform Reporter: Christopher Wawak <cwawak>
Component: InstallerAssignee: Etienne Simard <esimard>
Installer sub component: openshift-installer QA Contact: To Hung Sze <tsze>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: esimard, mstaeble, tsze
Version: 4.7   
Target Milestone: ---   
Target Release: 4.8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Private clusters deployed with IPI on Azure had an inbound NSG rule allowing SSH from any to any. Consequence: While SSH was not accessible from the Internet, this NSG rule could trigger an Azure security policy. Fix: Remove the inbound NSG rule for SSH on Azure private clusters. Result: That type of Azure security policy should not be triggered anymore for allowing SSH on the Internet. SSH is still allowed on the VNET through a default Azure NSG rule as designed.
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-07-27 22:55:38 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Christopher Wawak 2021-03-25 15:20:41 UTC
This bug was initially created as a copy of Bug #1943175

I am copying this bug to isolate SSH issue out of the other bug.


Version:
4.7.2, also 4.6.something

Platform:
azure

Please specify:
* IPI

What happened?
Customer reports unable to install IPI PRIVATE OpenShift cluster in Azure. This previously worked, but when certain policies were applied to the customer's Azure account, it stopped working.

The installer breaks on:

1. Unable to create network rule to allow SSH in.
    - "policyDefinition":{"name":"SSH access from the Internet should be blocked with Deny...

What did you expect to happen?
Installer completes successfully.

How to reproduce it (as minimally and precisely as possible)?
openshift-install using attached instsll-conmfig.yaml

Anything else we need to know?
- Installer log output attached
- This is a private cluster, so we shouldn't be failing on the SSH probe.

Comment 3 Etienne Simard 2021-03-25 20:45:05 UTC
Hello Christopher,

When deploying a cluster with the IPI on Azure, a temporary NSG rule is created to allow SSH to the bootstrap node. When you create a private cluster, the rule still exists, but you can't reach the bootstrap from the Internet (even when using the default configuration of "outboundType: Loadbalancer", as only Outbound NAT is enabled).


The temporary NSG rule and the Bootstrap machine are destroyed as soon the Bootstrap process is finished. Because no port is exposed to the Internet with a private cluster, you can only SSH to the bootstrap from the internal network.


I believe the reason your policy is flagging this is because the temporary NSG rule uses "Any" as the source. 


Thank you for the feedback, I will investigate how we can improve that NSG rule on private clusters.

Comment 4 Christopher Wawak 2021-03-26 11:45:25 UTC
My customer made the following suggestion:

> As a bug fix would it be possible for the NSG rule to set the source as the VNET since the VNET name is already provided during private cluster installs?

In that case, I suspect instead of being "Any", source as VNET wouldn't trigger at least that customer rule. Something to consider? Thanks!

Comment 6 Etienne Simard 2021-04-21 15:02:43 UTC
(In reply to Christopher Wawak from comment #4)
> My customer made the following suggestion:
> 
> > As a bug fix would it be possible for the NSG rule to set the source as the VNET since the VNET name is already provided during private cluster installs?
> 
> In that case, I suspect instead of being "Any", source as VNET wouldn't
> trigger at least that customer rule. Something to consider? Thanks!

Hello Christopher,

Thank you for the suggestion. The intended design for private clusters regarding ssh access to the bootstrap node is to only be able ssh to the bootstrap on the local network where it is deployed, which is what we're targeting on this bugzilla.

Comment 8 To Hung Sze 2021-05-07 15:42:31 UTC
I got an ipi private cluster up with bootstrap.
There is no public ip address for the bootstrap.
Is there somewhere else I should check?
Thanks.

Comment 9 Etienne Simard 2021-05-07 16:02:13 UTC
(In reply to To Hung Sze from comment #8)
> I got an ipi private cluster up with bootstrap.
> There is no public ip address for the bootstrap.
> Is there somewhere else I should check?
> Thanks.

The expected behavior of this bug fix is that when you create a private cluster on Azure, the "bootstrap_ssh_in" NSG rule will not exist. That's what should be verified.

Before the bug fix, this rule still existed on private clusters until the bootstrap was destroyed. It only affects the NSG rule (which would help with the security policy issue the user was running into).
There was already no possible way to access SSH from the Internet on a IPI Azure private cluster.

Comment 10 To Hung Sze 2021-05-11 15:44:51 UTC
I confirm, for internal cluster on Az,

with 4.7, there is a rule that allows ssh
Priority   Name               Port   Protocol   Source   Destination   Action
103        bootstrap_ssh_in   22     Tcp        Any      Any           Allow

I can ssh into the bootstrap that has an external IP address via tszeaz051121b-prxnl-bootstrap-pip-v4.


with 4.8, there is no such rule / external IP address.

Marking as verified.

Comment 13 errata-xmlrpc 2021-07-27 22:55:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438