Note this insane thing:

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:1b:97:12:95:47 brd ff:ff:ff:ff:ff:ff
    inet 192.0.2.118/24 brd 192.0.2.255 scope global dynamic eth0
       valid_lft 70sec preferred_lft 70sec
    inet6 fe80::21b:97ff:fe12:9547/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:8d:23:4d brd ff:ff:ff:ff:ff:ff
    inet 192.0.2.118 peer 192.0.2.1/32 scope global eth1
       valid_lft forever preferred_lft forever

that does not look right - the same IP address.

Would be really cool to get output of $ sudo journalctl -u openstack-ironic-discoverd-dnsmasq if it's still possible.

We've seen one more case of the same issue, where I was able to investigate more. Looked like the correct MAC did PXE booting, but then somehow it turned out that IP address was assigned to incorrect MAC. BOOTIF kernel parameter on a node also contained MAC address of incorrect MAC. No surprise that the ramdisk could not reach discoverd. Magic...

Any way around to cater this, we are hitting this too.

Created attachment 1049413 [details]
output of /log file

Created attachment 1049414 [details]
console

(In reply to Ola Pavlenko from comment #8)
> Created attachment 1049414 [details]
> console

output of journalctl

Created attachment 1049415 [details]
console 1

Whilst it seems as though libvirt and iPXE consistently order the NICs based on their PCI bus ID, i.e. first nic being net0, second net1, etc, the use of BOOTIF=${mac} is problematic when you have multiple NICs.

In my testing, what does seem to be consistent is that 'net0' is always the one that gets assigned the BOOTIF MAC address, which is not what we want if net1 is the pxe/provisioning network.

If you're sure that net1 is your provisioning network you can do:

# sed -i 's/mac/net1\/mac/' /httpboot/discoverd.ipxe

To give you BOOTIF=${net1/mac}, but this assumes that you're using net1, and that you've already ensured that net1 is your boot priority.

I suspect that this is an iPXE issue where it doesn't correctly assign ${mac} as the booting interface.

Confirming this issue. I've been doing trying out discovery with a set of multi-NIC libvirt VMs on my test environment. For some reason, discoverd.ipxe provides my system with the 2nd NIC's mac address when it boots off the 1st NIC. This conflict causes the discovery image to fail and hit dracut. Workaround for this issue for me was adding BOOTIF=${net0/mac} for my environment.

However, can't confirm if this affects a bare metal/IPMI setup. Someone might need to test this out on BM.

*** Bug 1249199 has been marked as a duplicate of this bug. ***

(In reply to Rhys Oxenham from comment #13)
> Whilst it seems as though libvirt and iPXE consistently order the NICs based
> on their PCI bus ID, i.e. first nic being net0, second net1, etc, the use of
> BOOTIF=${mac} is problematic when you have multiple NICs.
> 
> In my testing, what does seem to be consistent is that 'net0' is always the
> one that gets assigned the BOOTIF MAC address, which is not what we want if
> net1 is the pxe/provisioning network.
> 
> If you're sure that net1 is your provisioning network you can do:
> 
> # sed -i 's/mac/net1\/mac/' /httpboot/discoverd.ipxe
> 
> To give you BOOTIF=${net1/mac}, but this assumes that you're using net1, and
> that you've already ensured that net1 is your boot priority.
> 
> I suspect that this is an iPXE issue where it doesn't correctly assign
> ${mac} as the booting interface.

One of my customer confirmed that this work around helps him get past the discovery process, but again hits the bug when the node is rebooted for overcloud deployment. Is there any work around to overcome this stage?

"Unfortunately the workaround works for the discvovery image, but if I try to deploy the overcloud the PXE will fail.
It looks like that he starts booting from the 1st NIC, as I have set it in the Boot Options, but then he tried to get pxelinux.cfg for the 2nd NIC ??

I have attached a screenshot."

Ack on Dan's comments above, pxe_ssh really needs to be able to set the iPXE BOOTIF flag based on the NIC order of the virtual machine, or we hard code the requirement for the provisioning NIC to ALWAYS be on eth0/net0 and forcefully ensure that we use, say, BOOTIF=${net0/mac}.

The same is with deploy too; ${mac} also gets used there to uniquely identify the node, and deployments also fail. My hacky workaround is as follows, but works 100% of the time in my experience:


----
undercloud# cat << EOF > /usr/bin/bootif-fix
#!/usr/bin/env bash

while true;
        do find /httpboot/ -type f ! -iname "kernel" ! -iname "ramdisk" ! -iname "*.kernel" ! -iname "*.ramdisk" -exec sed -i 's|{mac|{net0/mac|g' {} +;
done
EOF

undercloud# chmod a+x /usr/bin/bootif-fix
undercloud# cat << EOF > /usr/lib/systemd/system/bootif-fix.service
[Unit]
Description=Automated fix for incorrect iPXE BOOFIF

[Service]
Type=simple
ExecStart=/usr/bin/bootif-fix

[Install]
WantedBy=multi-user.target
EOF

undercloud# systemctl daemon-reload
undercloud# systemctl enable bootif-fix
ln -s '/usr/lib/systemd/system/bootif-fix.service' '/etc/systemd/system/multi-user.target.wants/bootif-fix.service'
undercloud# systemctl start bootif-fix

------

Here is the hacky workaround that Rhys suggested in ready to copy-and-paste form. Make sure you are logged in as root and copy and paste this blob:

#####################################
cat << EOF > /usr/bin/bootif-fix
#!/usr/bin/env bash

while true;
        do find /httpboot/ -type f ! -iname "kernel" ! -iname "ramdisk" ! -iname "*.kernel" ! -iname "*.ramdisk" -exec sed -i 's|{mac|{net0/mac|g' {} +;
done
EOF

chmod a+x /usr/bin/bootif-fix

cat << EOF > /usr/lib/systemd/system/bootif-fix.service
[Unit]
Description=Automated fix for incorrect iPXE BOOFIF

[Service]
Type=simple
ExecStart=/usr/bin/bootif-fix

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable bootif-fix
systemctl start bootif-fix

#######################################

(In reply to Dan Sneddon from comment #22)

The above hacky workaround does seem to work in the case of virt. I have not (yet) been able to confirm that this workaround works in bare metal. In my test (on Juniper EX4200 switches configured for LACP bonds) it did not work.

regarding the doc-text, systemd will not only run the script at boot, but rather in a tight loop...

Any progress on this?

(In reply to Felipe Alfaro Solana from comment #27)
> Any progress on this?

Apply Comment 22 like work around

Any chance that manually patching ironic/drivers/modules/ipxe_config.template would help? I mean, instead of using the ugly "while true" loop?

(In reply to Edu Alcaniz from comment #28)
> (In reply to Felipe Alfaro Solana from comment #27)
> > Any progress on this?
> 
> Apply Comment 22 like work around

Yeah, I mean something that doesn't peg the CPU all the time :-)

(In reply to Felipe Alfaro Solana from comment #39)
> (In reply to Edu Alcaniz from comment #28)
> > (In reply to Felipe Alfaro Solana from comment #27)
> > > Any progress on this?
> > 
> > Apply Comment 22 like work around
> 
> Yeah, I mean something that doesn't peg the CPU all the time :-)

Patching the ipxe template should work just fine too.

I'm using the following:

    sed -i 's|${mac}|${net0/mac}|g' \
         /usr/share/instack-undercloud/ironic-discoverd/os-apply-config/httpboot/discoverd.ipxe \
         /usr/lib/python2.7/site-packages/ironic/drivers/modules/boot.ipxe \
         /usr/lib/python2.7/site-packages/ironic/drivers/modules/ipxe_config.template \
         /httpboot/*.ipxe

When we fix this bug (no matter how it gets fixed), we will need an option for choosing which NIC gets used. Most of the time, this will be net0, but it could just as easily be another NIC that should be used.

Hi,

If I get it correctly the problem is that the macro ${mac} in the iPXE template is pointing to the wrong mac address right? I've seem it and talked to iPXE developers about it, this problem has been fixed in iPXE since 2013 [1]

This has been fixed in Ironic as well [2], but the fix won't apply to discoverd since it uses a default iPXE configuration for booting all the machines.

I believe the right fix for this is just to use a new iPXE ROM, we should update the ipxe-bootimgs package (which is on the way I believe).

In the meantime, can you guys try fixing this by just replace the undionly.kpxe in the /tftpboot directory with a newer version?

You can get a pre-built one from http://boot.ipxe.org/undionly.kpxe

[1] http://git.ipxe.org/ipxe.git/commitdiff/66ea458
[2] https://bugs.launchpad.net/ironic/+bug/1504482

Rhys, Are you interested in trying this out?

Lucas, is this going to land for our next release based on liberty?

Hi @Chris,

The fix for this problem is in the ipxe-bootimgs package for RHEL/Centos which contains a old and buggy iPXE ROM, the BZ where the package rebase is being tracked is this one [0]. Once that's fixed this bug will go away.

The workarounds we currently have is to get a newer pre-built iPXE ROM from the fedora package or directly from the iPXE project [1] or build one locally [2]

[0] https://bugzilla.redhat.com/show_bug.cgi?id=1267030
[1] http://boot.ipxe.org/undionly.kpxe
[2] http://ipxe.org/download#source_code

Hi Lucas,

it was reported that pre-built one using the latest undionly.kpxe from boot.ipxe.org fails back to booting from localdisk with model Dell PowerEdge R720.

Perhaps I misunderstand something but would the mapping/orderings here work better if we used biosdevname (which should give predicable NIC ordering) in both the discovery ramdisk and the overcloud images? So long as NICs are ordered the same this should make things happier right?

This is not fixed in virt environments using the ipxe-bootimgs-20150821-1.git4e03af8e.el7.noarch.rpm package from https://bugzilla.redhat.com/show_bug.cgi?id=1267030

It does not work with http://boot.ipxe.org/undionly.kpxe either.

The only way I've gotten it to work is with a slightly modified command from c41:
sed -i 's|${mac|${net0/mac|g' \
  /usr/share/instack-undercloud/ironic-discoverd/os-apply-config/httpboot/discoverd.ipxe \
  /usr/lib/python2.7/site-packages/ironic/drivers/modules/boot.ipxe \
  /usr/lib/python2.7/site-packages/ironic/drivers/modules/ipxe_config.template \
  /httpboot/*.ipxe

With the trailing } the line didn't get update in boot.ipxe (${mac:hexhyp}), and once passed discovery the installation fails because the vm's can't deploy.

Jason, could you please provide us an environment for testing it? I'd ask the iPXE guy to take a look.

Hi Jason
> This is not fixed in virt environments using the
> ipxe-bootimgs-20150821-1.git4e03af8e.el7.noarch.rpm package from
> https://bugzilla.redhat.com/show_bug.cgi?id=1267030
> 
> It does not work with http://boot.ipxe.org/undionly.kpxe either.
> 

Yes, that's expected for virtual environments. QEMU already have iPXE embedded in the NICs (installed by the "ipxe-roms-qemu" package) so, updating ipxe-bootimags or copying the one from http://boot.ipxe.org to the TFTP directory will not work*.

* It doesn't work because we will chainload to the ROM image from the TFTP directory *only* if the node booting is booting from a standard PXE image. If it already boots from an iPXE image, as it happens on virtual environments, there's no chainloading. The flow for *non-virtual* environments goes like this:

1. Node boots with standard PXE and issue a DHCPREQUEST
2. DHCP server detects it and instruct PXE to fetch the iPXE image from the TFTP server and load it
3. Node boots with iPXE and issue a DHCPREQUEST
4. DHCP server now sees the request is coming from iPXE and return the iPXE script to boot the kernel/ramdisk and proceed with the deployment.

For virtual enviroments, since it's already booting from iPXE, the boot starts at the step 3. So the newer iPXE will never be fetched from the TFTP directory.

> The only way I've gotten it to work is with a slightly modified command from
> c41:
> sed -i 's|${mac|${net0/mac|g' \
>  
> /usr/share/instack-undercloud/ironic-discoverd/os-apply-config/httpboot/
> discoverd.ipxe \
>   /usr/lib/python2.7/site-packages/ironic/drivers/modules/boot.ipxe \
>  
> /usr/lib/python2.7/site-packages/ironic/drivers/modules/ipxe_config.template
> \
>   /httpboot/*.ipxe
> 
> With the trailing } the line didn't get update in boot.ipxe (${mac:hexhyp}),
> and once passed discovery the installation fails because the vm's can't
> deploy.

Right, yeah, by doing that you are hardcoding the use of the first NIC (net0). That seems a valid way of workarounding this problem while we don't have any a new "ipxe-roms-qemu" package to solve the problem for virtual environments.

One thing tho... The Ironic services (not discoverd) allows the user to set a custom script via it's configuration file so you don't have to change the default ones that will be overwritten again if there's an update in the package (because realistically, that's something people will have to do to adapt their environment).

To do that, you can change two configuration options under /etc/ironic/ironic.conf

[pxe]
ipxe_boot_script=<path to your custom boot.ipxe script>
pxe_config_template=<path to your custom ipxe_config.template>

(restart the openstack-ironic-conductor service after the changes)

Unfortunately for discoverd you will need to customize it at the /httpboot directory directly.

Hope that helps,
Lucas

Moving this to MODIFIED per Lucas -- the problem should be fixed on bare metal by the IPXE RPM we are now shipping with RHEL-OSP 8.

Would an updated ipxe-roms-qemu package fix the problem with virt environments? 

I am more than happy to file bugs against RHEL and Fedora in order to get them updated sooner rather than later.

Introspection completed ok on VMs with 2 nics.

Im running into the same issue on HW. One set of hardware runs through introspection fine, while the other does not. 

I have 3 supermicro servers with 6 nics
I have 10 dell m610's with 4 nics

The supermicro servers work fine
The dell servers boot from ipxe fine, however it tries to bring up the first nic, which is not the nic that is connected to the pxe network. I cannot complete introspection on the dell machines because they keep saying the network cannot be found

Hi Donny,

What OSP-d version are you running? The initial report for this ticket was for a virtual environment and I wasn't able to reproduce it in version 8 with VMs with multiple nics. 

Can you also check BZ#1301694 and see if matches the issue you're seeing.This was another introspection timeout related issue that came up only on HW during the 7.3 testing. 

Thanks

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0604.html

Hi,

I'm actually hitting this issue with virtualized controllers with 2 virtio NIC. I'm using the last OSP-d 8. Let me know which kind of information you need to help debug (but please be aware that it's a customer deployment so I'll not be able to connect to this env at some point!)

By the way, I though the find/sed loop given was using to much resources and was going through unrelated files so I switch it to:

#!/usr/bin/env bash

while true;
        do find /httpboot/ -type f \( -name "boot.ipxe" -o -name "inspector.pxe" -o -name "config" \) -exec sed -i 's|{mac|{net0/mac|g' {} +;
done

In my current deployment the undercloud is hosted on a very slow server (customer's choice, not mine obviously) and with the previous loop the undercloud VM was getting stuck at some point.

*** Bug 1326481 has been marked as a duplicate of this bug. ***