Created attachment 1146077 [details]
introspection logs - v7.3.1

Description of problem:

When attempting to perform node introspection, it fails ~75% of the time.

The error in logs is the following:

pr 11 10:35:16 ops2 dnsmasq-dhcp[1490]: DHCPREQUEST(br-ctlplane) 10.3.3.70 ec:f4:bb:e7:06:cc
Apr 11 10:35:16 ops2 dnsmasq-dhcp[1490]: DHCPACK(br-ctlplane) 10.3.3.70 ec:f4:bb:e7:06:cc
Apr 11 10:35:16 ops2 dnsmasq-dhcp[1490]: DHCPREQUEST(br-ctlplane) 10.3.3.70 ec:f4:bb:e7:06:cc
Apr 11 10:35:16 ops2 dnsmasq-dhcp[1490]: DHCPACK(br-ctlplane) 10.3.3.70 ec:f4:bb:e7:06:cc
Apr 11 10:35:16 ops2 dnsmasq-tftp[1490]: error 0 TFTP Aborted received from 10.3.3.70
Apr 11 10:35:16 ops2 dnsmasq-tftp[1490]: failed sending /tftpboot/undionly.kpxe to 10.3.3.70
Apr 11 10:35:16 ops2 dnsmasq-tftp[1490]: sent /tftpboot/undionly.kpxe to 10.3.3.70
Apr 11 10:35:17 ops2 ironic-api: 10.3.3.1 - - [11/Apr/2016 10:35:17] "GET / HTTP/1.0" 200 354

(more logs attached)

This causes iPXE boot to fail (and continue to boot from an old deployment on disk.)


Version-Release number of selected component (if applicable):

[stack@ops2 ~]$ rpm -qa | grep 'ironic\|tripleo\|oscplug' | sort
openstack-ironic-api-2015.1.2-2.el7ost.noarch
openstack-ironic-common-2015.1.2-2.el7ost.noarch
openstack-ironic-conductor-2015.1.2-2.el7ost.noarch
openstack-ironic-discoverd-1.1.0-8.el7ost.noarch
openstack-tripleo-0.0.7-0.1.1664e566.el7ost.noarch
openstack-tripleo-common-0.0.1.dev6-6.git49b57eb.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-123.el7ost.noarch
openstack-tripleo-image-elements-0.9.6-10.el7ost.noarch
openstack-tripleo-puppet-elements-0.0.1-5.el7ost.noarch
python-ironicclient-0.5.1-12.el7ost.noarch
python-ironic-discoverd-1.1.0-8.el7ost.noarch
python-rdomanager-oscplugin-0.0.10-28.el7ost.noarch


How reproducible:

75% of the time

Steps to Reproduce:
1. Open a console on the node to be inspected, then run the following script

### inspect-node.sh ####
#!/bin/bash
#
# pass in the node uuid as parameter 1 on the cli
uuid=$1

    echo "Starting introspection on $uuid"
    ironic node-set-maintenance $uuid true
    openstack baremetal introspection start $uuid
    RET_VAL=1
    while [ $RET_VAL -ne 0 ]; 
    do
      sleep 120
      ironic node-list | grep "${uuid}.*power off" &> /dev/null
      RET_VAL=$?
      if [ $RET_VAL -eq 1 ]; then
        echo "still running, sleeping..."
      fi
    done
    echo "introspection is complete. exiting"
    ironic node-set-maintenance $uuid false

2. Run 'journalctl -l -u openstack-ironic-discoverd -u openstack-ironic-discoverd-dnsmasq -u openstack-ironic-conductor -f' in a separate terminal
2. Get ready to hit ctl-b to drop to iPXE> prompt on the console
3. execute the script

Actual results:

see error above

Expected results:

success!

Additional info: