Bug 1326481

Virtual environment with pxe_ssh driver and 3 NICs of same type (all virtio, or all e1000)

Running introspection with the latest images from puddle:
http://rhos-release(...)/puddle-images/latest-8.0-images/
[ ]	CHECKSUM 	2016-04-12 12:47 	303 	 
[ ]	ironic-python-agent.tar	2016-04-12 12:03 	333M	 
[ ]	overcloud-full.tar 	2016-04-12 12:34 	1.0G	 
[ ]	version 	2016-04-12 12:46 	15 	

Introspection fails with Error 111 Connection refused.

I can ping and SSH to the instance: 
~~~
[root@undercloud ramdisk]# ping 192.0.2.102
PING 192.0.2.102 (192.0.2.102) 56(84) bytes of data.
64 bytes from 192.0.2.102: icmp_seq=1 ttl=64 time=0.332 ms
^C
--- 192.0.2.102 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.332/0.332/0.332/0.000 ms
[root@undercloud ramdisk]# ssh 192.0.2.102
The authenticity of host '192.0.2.102 (192.0.2.102)' can't be established.
ECDSA key fingerprint is fd:21:9d:f8:0a:5b:08:4e:33:d3:7d:44:4b:a7:4e:5a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.0.2.102' (ECDSA) to the list of known hosts.
root.2.102's password: 

[root@undercloud ramdisk]# 
~~~

ARP table logically gets populated:
~~~
[root@undercloud ramdisk]# arp -n | grep 102
192.0.2.102              ether   52:54:00:69:bf:c7   C                     br-ctlplane
~~~


Switching the interface eth0 from virtio to e1000 and keeping eth1 and eth2 as virtio makes introspection work successfully:
~~~
[stack@undercloud ~]$ ironic node-list | awk '{print $2}' | xargs -I {} ironic node-delete {}
Invalid input for field/attribute node_ident. Value: 'UUID'. unable to convert to uuid_or_name (HTTP 400)
Deleted node f6f2ef34-b943-4cc9-90e0-29e6b9565e9c
Deleted node 20b21f74-68bb-4262-a6f7-cdfa783f4f83
Deleted node 177341e4-9222-464a-ad7b-814043914cf2
[stack@undercloud ~]$ openstack baremetal import --json instackenv.ceph.json 
[stack@undercloud ~]$ ironic node-list
+--------------------------------------+-----------------+---------------+-------------+--------------------+-------------+
| UUID                                 | Name            | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+-----------------+---------------+-------------+--------------------+-------------+
| 83ba1391-d596-4651-9b58-b1258ebb0652 | overcloud-ceph1 | None          | power off   | available          | False       |
| 379e4486-d9f9-46d5-9e68-b332257c89e0 | overcloud-ceph2 | None          | power off   | available          | False       |
| b9eb9a38-1fe0-4b8f-bd2d-806c8e4a7988 | overcloud-ceph3 | None          | power off   | available          | False       |
+--------------------------------------+-----------------+---------------+-------------+--------------------+-------------+
[stack@undercloud ~]$ openstack baremetal introspection bulk start
Setting nodes for introspection to manageable...
Starting introspection of node: 83ba1391-d596-4651-9b58-b1258ebb0652
Starting introspection of node: 379e4486-d9f9-46d5-9e68-b332257c89e0
Starting introspection of node: b9eb9a38-1fe0-4b8f-bd2d-806c8e4a7988
Waiting for introspection to finish...
Introspection for UUID 379e4486-d9f9-46d5-9e68-b332257c89e0 finished successfully.
~~~

Switching all interfaces to e1000 does not work. All interface with virtio does not work.

[root@rhospbl-5 ~]# virsh list --all | grep ceph | awk '{print $2}' | while read node;do echo -e "\n\n$node";echo "===="; virsh dumpxml $node | grep mac -A2;done


overcloud-ceph1
====
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
    <boot dev='network'/>
  </os>
  <features>
--
      <mac address='52:54:00:dd:93:83'/>
      <source network='provisioning' bridge='virbr2'/>
      <target dev='vnet18'/>
      <model type='virtio'/>
--
      <mac address='52:54:00:e8:bb:79'/>
      <source network='trunk' bridge='virbr1'/>
      <target dev='vnet19'/>
      <model type='virtio'/>
--
      <mac address='52:54:00:44:ea:6f'/>
      <source bridge='br0'/>
      <target dev='vnet20'/>
      <model type='virtio'/>


overcloud-ceph3
====
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
    <boot dev='network'/>
  </os>
  <features>
--
      <mac address='52:54:00:13:97:cb'/>
      <source network='provisioning' bridge='virbr2'/>
      <target dev='vnet24'/>
      <model type='e1000'/>
--
      <mac address='52:54:00:cb:79:5c'/>
      <source network='trunk' bridge='virbr1'/>
      <target dev='vnet25'/>
      <model type='e1000'/>
--
      <mac address='52:54:00:55:be:d1'/>
      <source network='redhat' bridge='br0'/>
      <target dev='vnet26'/>
      <model type='e1000'/>


overcloud-ceph2
====
    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
    <boot dev='network'/>
  </os>
  <features>
--
      <mac address='52:54:00:c4:aa:1a'/>
      <source network='provisioning'/>
      <model type='e1000'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
--
      <mac address='52:54:00:5b:9d:f7'/>
      <source network='trunk'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
--
      <mac address='52:54:00:38:ef:38'/>
      <source network='redhat'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
[root@rhospbl-5 ~]# 


I can reproduce this in any combination (make ceph0, ceph1 or ceph2 work simply by changing network types).