Bug 1886525 - [Baremetal IPI ]Ironic fails to Power on nodes when Proxy is included in install-config.yaml
Summary: [Baremetal IPI ]Ironic fails to Power on nodes when Proxy is included in inst...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Bare Metal Hardware Provisioning
Version: 4.5
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: ---
: 4.7.0
Assignee: Stephen Benjamin
QA Contact: Amit Ugol
URL:
Whiteboard:
Depends On:
Blocks: dit
TreeView+ depends on / blocked
 
Reported: 2020-10-08 16:36 UTC by DirectedSoul
Modified: 2020-10-15 13:47 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Dell PowerEdge R720
Last Closed: 2020-10-15 13:47:15 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description DirectedSoul 2020-10-08 16:36:32 UTC
Version:

$ openshift-install version

4.5.13 

Platform:

BareMetal IPI 

Please specify:
* IPI (automated install with `openshift-install`.

What happened?

Ironic fails to Power on my master nodes when proxy(squid) is included in deployment process. The same set up works without enabling proxy. 

From the `ironic-conductor container`, specifically below (I'm attaching the passed ironic logs(without proxy usage) and failed ironic logs(with proxy usage at the end)

```
 MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/engines.py:307
2020-10-08 15:27:08.356 1 INFO ironic.conductor.base_manager [req-484cec0c-9ecf-498c-8c37-951af720dede - - - - -] Successfully started conductor with hostname localhost.localdomain.
2020-10-08 15:27:08.357 1 INFO ironic.common.rpc_service [req-484cec0c-9ecf-498c-8c37-951af720dede - - - - -] Created RPC server for service ironic.conductor_manager on host localhost.localdomain.
2020-10-08 15:28:03.539 1 DEBUG futurist.periodics [-] Submitting periodic callback 'ironic.conductor.manager.ConductorManager._check_cleanwait_timeouts' _process_scheduled /usr/lib/python3.6/site-packages/futurist/periodics.py:639
2020-10-08 15:28:03.540 1 DEBUG futurist.periodics [-] Submitting periodic callback 'ironic.conductor.manager.ConductorManager._check_deploy_timeouts' _process_scheduled /usr/lib/python3.6/site-packages/futurist/periodics.py:639
2020-10-08 15:28:03.541 1 DEBUG futurist.periodics [-] Submitting periodic callback 'ironic.conductor.manager.ConductorManager._check_inspect_wait_timeouts' _process_scheduled /usr/lib/python3.6/site-packages/futurist/periodics.py:639
2020-10-08 15:28:03.541 1 DEBUG futurist.periodics [-] Submitting periodic callback 'ironic.conductor.manager.ConductorManager._check_orphan_allocations' _process_scheduled /usr/lib/python3.6/site-packages/futurist/periodics.py:639

```

Im attaching the `install-config.yaml` contents which has the `proxy` element

```
apiVersion: v1
baseDomain: cloud.lab.eng.bos.redhat.com
proxy:
  httpProxy: http://10.19.141.190:3128/
  httpsProxy: http://10.19.141.190:3128/
  noProxy: 172.22.0.0/24, 10.19.0.0/16
metadata:
  name: kni3
networking:
  machineCIDR: 10.19.3.128/26
  networkType: OVNKubernetes
```


What did you expect to happen?

Expecting a successful install when proxy is used during installation. 

How to reproduce it (as minimally and precisely as possible)?

Add the proxy in the install-config.yaml file in the `noProxy` field also include your subnet. 

```
apiVersion: v1
baseDomain: cloud.lab.eng.bos.redhat.com
proxy:
  httpProxy: http://10.19.141.190:3128/
  httpsProxy: http://10.19.141.190:3128/
  noProxy: 172.22.0.0/24, 10.19.0.0/16 

```

Anything else we need to know?

* Failed ironic-conductor.log 

Im happy to provide any other logs needed.

Comment 4 Stephen Benjamin 2020-10-14 15:46:44 UTC
Removing triaged keyword so it gets reviewed by developers, and setting against Ironic to have them look

Comment 7 Stephen Benjamin 2020-10-14 16:05:27 UTC
The logs show the installer isn't reaching the Ironic API, so the problem is communication from the installer itself to the bootstrap host.  Can you confirm the environment the installer is being executed from? This is different than the install-config, this is the environment that is used to run the command.

So, if the installer doesn't need a proxy to talk to the bootstrap API then that needs to be in NO_PROXY. It's also tricky to get right, since how proxy variables are handled varies between software.

Can you share `env` output from where you're running the installer?

Comment 9 Derek Higgins 2020-10-15 13:47:15 UTC
This will be documented in the baremetal-deploy docs.


Note You need to log in before you can comment on or make changes to this bug.