Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1886525

Summary:	[Baremetal IPI ]Ironic fails to Power on nodes when Proxy is included in install-config.yaml
Product:	OpenShift Container Platform	Reporter:	DirectedSoul <shegde>
Component:	Bare Metal Hardware Provisioning	Assignee:	Stephen Benjamin <stbenjam>
Bare Metal Hardware Provisioning sub component:	ironic	QA Contact:	Amit Ugol <augol>
Status:	CLOSED NOTABUG	Docs Contact:
Severity:	medium
Priority:	medium	CC:	derekh, pablo.iranzo, rlopez, shegde
Version:	4.5
Target Milestone:	---
Target Release:	4.7.0
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:	Dell PowerEdge R720
Last Closed:	2020-10-15 13:47:15 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1831748

Description DirectedSoul 2020-10-08 16:36:32 UTC

Version:

$ openshift-install version

4.5.13 

Platform:

BareMetal IPI 

Please specify:
* IPI (automated install with `openshift-install`.

What happened?

Ironic fails to Power on my master nodes when proxy(squid) is included in deployment process. The same set up works without enabling proxy. 

From the `ironic-conductor container`, specifically below (I'm attaching the passed ironic logs(without proxy usage) and failed ironic logs(with proxy usage at the end)

```
 MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/engines.py:307
2020-10-08 15:27:08.356 1 INFO ironic.conductor.base_manager [req-484cec0c-9ecf-498c-8c37-951af720dede - - - - -] Successfully started conductor with hostname localhost.localdomain.
2020-10-08 15:27:08.357 1 INFO ironic.common.rpc_service [req-484cec0c-9ecf-498c-8c37-951af720dede - - - - -] Created RPC server for service ironic.conductor_manager on host localhost.localdomain.
2020-10-08 15:28:03.539 1 DEBUG futurist.periodics [-] Submitting periodic callback 'ironic.conductor.manager.ConductorManager._check_cleanwait_timeouts' _process_scheduled /usr/lib/python3.6/site-packages/futurist/periodics.py:639
2020-10-08 15:28:03.540 1 DEBUG futurist.periodics [-] Submitting periodic callback 'ironic.conductor.manager.ConductorManager._check_deploy_timeouts' _process_scheduled /usr/lib/python3.6/site-packages/futurist/periodics.py:639
2020-10-08 15:28:03.541 1 DEBUG futurist.periodics [-] Submitting periodic callback 'ironic.conductor.manager.ConductorManager._check_inspect_wait_timeouts' _process_scheduled /usr/lib/python3.6/site-packages/futurist/periodics.py:639
2020-10-08 15:28:03.541 1 DEBUG futurist.periodics [-] Submitting periodic callback 'ironic.conductor.manager.ConductorManager._check_orphan_allocations' _process_scheduled /usr/lib/python3.6/site-packages/futurist/periodics.py:639

```

Im attaching the `install-config.yaml` contents which has the `proxy` element

```
apiVersion: v1
baseDomain: cloud.lab.eng.bos.redhat.com
proxy:
  httpProxy: http://10.19.141.190:3128/
  httpsProxy: http://10.19.141.190:3128/
  noProxy: 172.22.0.0/24, 10.19.0.0/16
metadata:
  name: kni3
networking:
  machineCIDR: 10.19.3.128/26
  networkType: OVNKubernetes
```


What did you expect to happen?

Expecting a successful install when proxy is used during installation. 

How to reproduce it (as minimally and precisely as possible)?

Add the proxy in the install-config.yaml file in the `noProxy` field also include your subnet. 

```
apiVersion: v1
baseDomain: cloud.lab.eng.bos.redhat.com
proxy:
  httpProxy: http://10.19.141.190:3128/
  httpsProxy: http://10.19.141.190:3128/
  noProxy: 172.22.0.0/24, 10.19.0.0/16 

```

Anything else we need to know?

* Failed ironic-conductor.log 

Im happy to provide any other logs needed.

Comment 4 Stephen Benjamin 2020-10-14 15:46:44 UTC

Removing triaged keyword so it gets reviewed by developers, and setting against Ironic to have them look

Comment 7 Stephen Benjamin 2020-10-14 16:05:27 UTC

The logs show the installer isn't reaching the Ironic API, so the problem is communication from the installer itself to the bootstrap host.  Can you confirm the environment the installer is being executed from? This is different than the install-config, this is the environment that is used to run the command.

So, if the installer doesn't need a proxy to talk to the bootstrap API then that needs to be in NO_PROXY. It's also tricky to get right, since how proxy variables are handled varies between software.

Can you share `env` output from where you're running the installer?

Comment 9 Derek Higgins 2020-10-15 13:47:15 UTC

This will be documented in the baremetal-deploy docs.