Bug 1317695 - Ironic Inspector should blacklist all MAC addresses on host after discovery
Summary: Ironic Inspector should blacklist all MAC addresses on host after discovery
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-ironic-inspector
Version: 8.0 (Liberty)
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ga
: 10.0 (Newton)
Assignee: Dmitry Tantsur
QA Contact: Alexander Chuzhoy
URL:
Whiteboard:
Depends On:
Blocks: 1326996
TreeView+ depends on / blocked
 
Reported: 2016-03-14 22:08 UTC by Dan Sneddon
Modified: 2019-10-10 11:32 UTC (History)
7 users (show)

Fixed In Version: openstack-ironic-inspector-2.2.5-2.el7ost.noarch
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1326996 (view as bug list)
Environment:
Last Closed: 2016-04-18 16:38:01 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
OpenStack gerrit 293475 None None None 2016-03-16 14:45:04 UTC
Launchpad 1557979 None None None 2016-03-16 09:50:38 UTC

Description Dan Sneddon 2016-03-14 22:08:29 UTC
Description of problem:
Some servers cannot disable PXE boot on secondary NIC interfaces, so after discovery and deployment, the overcloud node ends up booting the discovery image.

Version-Release number of selected component (if applicable):
OSP 7.3 + OSP 8

How reproducible:
100%

Steps to Reproduce:
1. Configure multiple NICs for PXE boot
2. Perform discovery
3. Perform deployment
4. Reboot overcloud servers

Actual results:
Nodes will go through PXE boot timeout on first NIC, then will boot via the second NIC the discovery image.

Expected results:
After deployment, the nodes should boot to disk.

Additional info:
Our customer are using blades from Lenovo and the
"plain" ipmi_tool driver for Ironic. The blades have four NICs and
somehow (at least this is what I've been told by the HW guys) it's not
possible to only enable PXE on a single interface, it's all or nothing.

Comment 2 Dmitry Tantsur 2016-03-15 10:40:13 UTC
Hello!

This request quite contradicts to how introspection actually works. E.g. how do you expect it to figure out that the introspection is done forever?

The simplest thing to do is:

 sudo systemctl stop openstack-ironic-inspector-dnsmasq
 sudo systemctl disable openstack-ironic-inspector-dnsmasq

If you want some TripleO command to do it (not sure if it's a great idea though), please redirect this bug to the generic project.

P.S.
When Nova is finally able to work with several MAC's per node, we'll just enroll all MAC's on the provisioning network, and avoid this problem.

Comment 3 Dan Sneddon 2016-03-15 17:39:20 UTC
(In reply to Dmitry Tantsur from comment #2)
> Hello!
> 
> This request quite contradicts to how introspection actually works. E.g. how
> do you expect it to figure out that the introspection is done forever?
> 
> The simplest thing to do is:
> 
>  sudo systemctl stop openstack-ironic-inspector-dnsmasq
>  sudo systemctl disable openstack-ironic-inspector-dnsmasq
> 
> If you want some TripleO command to do it (not sure if it's a great idea
> though), please redirect this bug to the generic project.
> 
> P.S.
> When Nova is finally able to work with several MAC's per node, we'll just
> enroll all MAC's on the provisioning network, and avoid this problem.

We do, in fact, know that introspection has been completed for a particular node. We already blacklist the MAC that was used to boot the system. I fail to see any downside to blacklisting the rest of the MACs on that same system.

I don't really understand why we would need Nova to support multiple MACs per instance. We only need to add the other MAC addresses to the iptables blacklist for dnsmasq. If those two things are somehow tied together, I don't know how.

Thanks for the workaround, but that would disable scaling. Then, in order to scale up computes, you would have to turn this service back on. If any nodes were to reboot during the time the service was enabled, they would not boot to disk.

Comment 4 Dmitry Tantsur 2016-03-16 09:47:56 UTC
> We do, in fact, know that introspection has been completed for a particular node. We already blacklist the MAC that was used to boot the system. I fail to see any downside to blacklisting the rest of the MACs on that same system.

The only problem is that we don't really know them, cause we only keep one MAC in Ironic.

That said, I got an idea which I'll try to bring upstream. Stay tuned.

Comment 5 Mike Burns 2016-04-07 21:14:44 UTC
This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.

Comment 6 Dmitry Tantsur 2016-04-08 07:55:23 UTC
I'm terribly sorry, I forgot to update the bug status. It did make it into OSP 8.0 and is available from puddles for some time.

Comment 11 Alexander Chuzhoy 2016-04-14 20:36:55 UTC
Verified:

Environment:
openstack-ironic-inspector-2.2.5-2.el7ost.noarch


Before introspecting the nodes:
[root@undercloud ~]# iptables -L ironic-inspector
Chain ironic-inspector (1 references)                    
target     prot opt source     destination                            
REJECT     all  --  anywhere anywhere   reject-with icmp-port-unreachable

When the introspection just started:
[root@undercloud ~]# iptables -L ironic-inspector
Chain ironic-inspector (1 references)                   
target     prot opt source d         estination   
DROP       all  --  anywhere             anywhere             MAC 00:0A:F7:7F:24:88            
DROP       all  --  anywhere             anywhere             MAC 00:0A:F7:7F:24:96            
DROP       all  --  anywhere             anywhere             MAC 00:0A:F7:79:93:CE            
DROP       all  --  anywhere             anywhere             MAC 00:0A:F7:79:93:1A            
DROP       all  --  anywhere             anywhere             MAC 00:0A:F7:79:93:2A            
DROP       all  --  anywhere             anywhere             MAC 00:0A:F7:7F:24:9E            
DROP       all  --  anywhere             anywhere             MAC 00:0A:F7:7F:24:5E            
ACCEPT     all  --  anywhere             anywhere 


Lated during the introspection:
[root@undercloud ~]# iptables -L ironic-inspector
Chain ironic-inspector (1 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere


After the introspection:
[root@undercloud ~]# iptables -L ironic-inspector
Chain ironic-inspector (1 references)
target     prot opt source               destination
REJECT     all  --  anywhere anywhere   reject-with icmp-port-unreachable



So as we see above, after discovery everything is blacklisted.


Note You need to log in before you can comment on or make changes to this bug.