Hide Forgot
Virtual environment with pxe_ssh driver and 3 NICs of same type (all virtio, or all e1000) Running introspection with the latest images from puddle: http://rhos-release(...)/puddle-images/latest-8.0-images/ [ ] CHECKSUM 2016-04-12 12:47 303 [ ] ironic-python-agent.tar 2016-04-12 12:03 333M [ ] overcloud-full.tar 2016-04-12 12:34 1.0G [ ] version 2016-04-12 12:46 15 Introspection fails with Error 111 Connection refused. I can ping and SSH to the instance: ~~~ [root@undercloud ramdisk]# ping 192.0.2.102 PING 192.0.2.102 (192.0.2.102) 56(84) bytes of data. 64 bytes from 192.0.2.102: icmp_seq=1 ttl=64 time=0.332 ms ^C --- 192.0.2.102 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.332/0.332/0.332/0.000 ms [root@undercloud ramdisk]# ssh 192.0.2.102 The authenticity of host '192.0.2.102 (192.0.2.102)' can't be established. ECDSA key fingerprint is fd:21:9d:f8:0a:5b:08:4e:33:d3:7d:44:4b:a7:4e:5a. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '192.0.2.102' (ECDSA) to the list of known hosts. root.2.102's password: [root@undercloud ramdisk]# ~~~ ARP table logically gets populated: ~~~ [root@undercloud ramdisk]# arp -n | grep 102 192.0.2.102 ether 52:54:00:69:bf:c7 C br-ctlplane ~~~ Switching the interface eth0 from virtio to e1000 and keeping eth1 and eth2 as virtio makes introspection work successfully: ~~~ [stack@undercloud ~]$ ironic node-list | awk '{print $2}' | xargs -I {} ironic node-delete {} Invalid input for field/attribute node_ident. Value: 'UUID'. unable to convert to uuid_or_name (HTTP 400) Deleted node f6f2ef34-b943-4cc9-90e0-29e6b9565e9c Deleted node 20b21f74-68bb-4262-a6f7-cdfa783f4f83 Deleted node 177341e4-9222-464a-ad7b-814043914cf2 [stack@undercloud ~]$ openstack baremetal import --json instackenv.ceph.json [stack@undercloud ~]$ ironic node-list +--------------------------------------+-----------------+---------------+-------------+--------------------+-------------+ | UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance | +--------------------------------------+-----------------+---------------+-------------+--------------------+-------------+ | 83ba1391-d596-4651-9b58-b1258ebb0652 | overcloud-ceph1 | None | power off | available | False | | 379e4486-d9f9-46d5-9e68-b332257c89e0 | overcloud-ceph2 | None | power off | available | False | | b9eb9a38-1fe0-4b8f-bd2d-806c8e4a7988 | overcloud-ceph3 | None | power off | available | False | +--------------------------------------+-----------------+---------------+-------------+--------------------+-------------+ [stack@undercloud ~]$ openstack baremetal introspection bulk start Setting nodes for introspection to manageable... Starting introspection of node: 83ba1391-d596-4651-9b58-b1258ebb0652 Starting introspection of node: 379e4486-d9f9-46d5-9e68-b332257c89e0 Starting introspection of node: b9eb9a38-1fe0-4b8f-bd2d-806c8e4a7988 Waiting for introspection to finish... Introspection for UUID 379e4486-d9f9-46d5-9e68-b332257c89e0 finished successfully. ~~~ Switching all interfaces to e1000 does not work. All interface with virtio does not work. [root@rhospbl-5 ~]# virsh list --all | grep ceph | awk '{print $2}' | while read node;do echo -e "\n\n$node";echo "===="; virsh dumpxml $node | grep mac -A2;done overcloud-ceph1 ==== <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type> <boot dev='network'/> </os> <features> -- <mac address='52:54:00:dd:93:83'/> <source network='provisioning' bridge='virbr2'/> <target dev='vnet18'/> <model type='virtio'/> -- <mac address='52:54:00:e8:bb:79'/> <source network='trunk' bridge='virbr1'/> <target dev='vnet19'/> <model type='virtio'/> -- <mac address='52:54:00:44:ea:6f'/> <source bridge='br0'/> <target dev='vnet20'/> <model type='virtio'/> overcloud-ceph3 ==== <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type> <boot dev='network'/> </os> <features> -- <mac address='52:54:00:13:97:cb'/> <source network='provisioning' bridge='virbr2'/> <target dev='vnet24'/> <model type='e1000'/> -- <mac address='52:54:00:cb:79:5c'/> <source network='trunk' bridge='virbr1'/> <target dev='vnet25'/> <model type='e1000'/> -- <mac address='52:54:00:55:be:d1'/> <source network='redhat' bridge='br0'/> <target dev='vnet26'/> <model type='e1000'/> overcloud-ceph2 ==== <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type> <boot dev='network'/> </os> <features> -- <mac address='52:54:00:c4:aa:1a'/> <source network='provisioning'/> <model type='e1000'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> -- <mac address='52:54:00:5b:9d:f7'/> <source network='trunk'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> -- <mac address='52:54:00:38:ef:38'/> <source network='redhat'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> [root@rhospbl-5 ~]# I can reproduce this in any combination (make ceph0, ceph1 or ceph2 work simply by changing network types).
Created attachment 1146618 [details] screenshot of fail
Created attachment 1146619 [details] e1000-virtio-virtio vs virtio-virtio-virtio
Created attachment 1146620 [details] fail
Created attachment 1146621 [details] fail
Created attachment 1146622 [details] fail
This seems to resemble: https://bugzilla.redhat.com/show_bug.cgi?id=1209176
[root@rhospbl-5 ~]# virsh net-dumpxml provisioning <network connections='3'> <name>provisioning</name> <uuid>de1d3965-2695-4a1d-9153-ba1e5c268e4e</uuid> <bridge name='virbr2' stp='on' delay='0'/> <mac address='52:54:00:ad:34:2f'/> </network> [root@rhospbl-5 ~]# virsh net-dumpxml trunk <network connections='4'> <name>trunk</name> <uuid>3858a7e6-d70f-4842-90f7-2d1b7fee51d9</uuid> <bridge name='virbr1' stp='on' delay='0'/> <mac address='52:54:00:8b:d4:fb'/> </network> [root@rhospbl-5 ~]# virsh net-dumpxml redhat <network connections='3'> <name>redhat</name> <uuid>882f0fc6-37d7-4368-a388-9ec5993e0baf</uuid> <forward mode='bridge'/> <bridge name='br0'/> </network>
Switching all interfaces to Nic1: e1000 Nic2: virtio Nic3: virtio makes introspection work [stack@undercloud ~]$ openstack baremetal introspection bulk start Setting nodes for introspection to manageable... Starting introspection of node: 5a484be8-4963-42fe-ba03-b1c228996ce6 Starting introspection of node: af984d74-67ca-4917-b760-5b313994b2a0 Starting introspection of node: 8a36b1dc-a626-414d-b601-cf7a29ba6a7e Starting introspection of node: 1d541fa2-36d1-43a6-a7f6-af7f1fe09838 Starting introspection of node: a5cdd130-f41c-42ec-a681-4807c8fd318a Starting introspection of node: b66b8835-993f-49fa-bb22-e93d3d6bc25f Starting introspection of node: 6652a3b1-3b30-4c99-b934-ed1b09d72b8d Starting introspection of node: 3b03d4fe-f478-4655-9ef6-f6a861e67c68 Waiting for introspection to finish... Introspection for UUID 5a484be8-4963-42fe-ba03-b1c228996ce6 finished successfully. Introspection for UUID 8a36b1dc-a626-414d-b601-cf7a29ba6a7e finished successfully. Introspection for UUID af984d74-67ca-4917-b760-5b313994b2a0 finished successfully. Introspection for UUID a5cdd130-f41c-42ec-a681-4807c8fd318a finished successfully. Introspection for UUID 6652a3b1-3b30-4c99-b934-ed1b09d72b8d finished successfully. Introspection for UUID b66b8835-993f-49fa-bb22-e93d3d6bc25f finished successfully. Introspection for UUID 3b03d4fe-f478-4655-9ef6-f6a861e67c68 finished successfully. Introspection for UUID 1d541fa2-36d1-43a6-a7f6-af7f1fe09838 finished successfully. Setting manageable nodes to available... Node 5a484be8-4963-42fe-ba03-b1c228996ce6 has been set to available. Node af984d74-67ca-4917-b760-5b313994b2a0 has been set to available. Node 8a36b1dc-a626-414d-b601-cf7a29ba6a7e has been set to available. Node 1d541fa2-36d1-43a6-a7f6-af7f1fe09838 has been set to available. Node a5cdd130-f41c-42ec-a681-4807c8fd318a has been set to available. Node b66b8835-993f-49fa-bb22-e93d3d6bc25f has been set to available. Node 6652a3b1-3b30-4c99-b934-ed1b09d72b8d has been set to available. Node 3b03d4fe-f478-4655-9ef6-f6a861e67c68 has been set to available. Introspection completed. [stack@undercloud ~]$
Created attachment 1146641 [details] xml of all vms
Created attachment 1146642 [details] ramdisk logs
Created attachment 1146643 [details] ramdisk logs
Created attachment 1146644 [details] ramdisk logs
Created attachment 1146645 [details] ramdisk logs
Created attachment 1146646 [details] ramdisk logs
Created attachment 1146647 [details] ram disk
Created attachment 1146648 [details] ramdisk logs
Created attachment 1146649 [details] ramdisk logs
Created attachment 1146650 [details] ramdisk logs
*** This bug has been marked as a duplicate of bug 1234601 ***