Bug 1326481 - Introspection fails with pxe_ssh in OSP 8, with 3 NICs of same type
Summary: Introspection fails with pxe_ssh in OSP 8, with 3 NICs of same type
Keywords:
Status: CLOSED DUPLICATE of bug 1234601
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-ironic-inspector
Version: 8.0 (Liberty)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: async
: ---
Assignee: Dmitry Tantsur
QA Contact: Raviv Bar-Tal
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-04-12 19:17 UTC by Andreas Karis
Modified: 2016-08-19 08:17 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-08-19 08:17:24 UTC
Target Upstream Version:


Attachments (Terms of Use)
screenshot of fail (57.12 KB, image/png)
2016-04-12 19:18 UTC, Andreas Karis
no flags Details
e1000-virtio-virtio vs virtio-virtio-virtio (110.01 KB, image/png)
2016-04-12 19:19 UTC, Andreas Karis
no flags Details
fail (68.41 KB, image/png)
2016-04-12 19:19 UTC, Andreas Karis
no flags Details
fail (73.16 KB, image/png)
2016-04-12 19:19 UTC, Andreas Karis
no flags Details
fail (79.09 KB, image/png)
2016-04-12 19:19 UTC, Andreas Karis
no flags Details
xml of all vms (34.89 KB, text/plain)
2016-04-12 19:47 UTC, Andreas Karis
no flags Details
ramdisk logs (13.21 MB, application/x-gzip)
2016-04-12 19:47 UTC, Andreas Karis
no flags Details
ramdisk logs (14.31 MB, application/x-gzip)
2016-04-12 19:48 UTC, Andreas Karis
no flags Details
ramdisk logs (115 bytes, application/x-gzip)
2016-04-12 19:52 UTC, Andreas Karis
no flags Details
ramdisk logs (7.80 MB, application/x-gzip)
2016-04-12 19:52 UTC, Andreas Karis
no flags Details
ramdisk logs (16.41 MB, application/x-gzip)
2016-04-12 19:53 UTC, Andreas Karis
no flags Details
ram disk (13.18 MB, application/x-gzip)
2016-04-12 20:01 UTC, Andreas Karis
no flags Details
ramdisk logs (2.84 MB, application/x-gzip)
2016-04-12 20:01 UTC, Andreas Karis
no flags Details
ramdisk logs (13.66 MB, application/x-gzip)
2016-04-12 20:02 UTC, Andreas Karis
no flags Details
ramdisk logs (12.87 MB, application/x-gzip)
2016-04-12 20:04 UTC, Andreas Karis
no flags Details

Description Andreas Karis 2016-04-12 19:17:07 UTC
Virtual environment with pxe_ssh driver and 3 NICs of same type (all virtio, or all e1000)

Running introspection with the latest images from puddle:
http://rhos-release(...)/puddle-images/latest-8.0-images/
[ ]	CHECKSUM 	2016-04-12 12:47 	303 	 
[ ]	ironic-python-agent.tar	2016-04-12 12:03 	333M	 
[ ]	overcloud-full.tar 	2016-04-12 12:34 	1.0G	 
[ ]	version 	2016-04-12 12:46 	15 	

Introspection fails with Error 111 Connection refused.

I can ping and SSH to the instance: 
~~~
[root@undercloud ramdisk]# ping 192.0.2.102
PING 192.0.2.102 (192.0.2.102) 56(84) bytes of data.
64 bytes from 192.0.2.102: icmp_seq=1 ttl=64 time=0.332 ms
^C
--- 192.0.2.102 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.332/0.332/0.332/0.000 ms
[root@undercloud ramdisk]# ssh 192.0.2.102
The authenticity of host '192.0.2.102 (192.0.2.102)' can't be established.
ECDSA key fingerprint is fd:21:9d:f8:0a:5b:08:4e:33:d3:7d:44:4b:a7:4e:5a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.0.2.102' (ECDSA) to the list of known hosts.
root.2.102's password: 

[root@undercloud ramdisk]# 
~~~

ARP table logically gets populated:
~~~
[root@undercloud ramdisk]# arp -n | grep 102
192.0.2.102              ether   52:54:00:69:bf:c7   C                     br-ctlplane
~~~


Switching the interface eth0 from virtio to e1000 and keeping eth1 and eth2 as virtio makes introspection work successfully:
~~~
[stack@undercloud ~]$ ironic node-list | awk '{print $2}' | xargs -I {} ironic node-delete {}
Invalid input for field/attribute node_ident. Value: 'UUID'. unable to convert to uuid_or_name (HTTP 400)
Deleted node f6f2ef34-b943-4cc9-90e0-29e6b9565e9c
Deleted node 20b21f74-68bb-4262-a6f7-cdfa783f4f83
Deleted node 177341e4-9222-464a-ad7b-814043914cf2
[stack@undercloud ~]$ openstack baremetal import --json instackenv.ceph.json 
[stack@undercloud ~]$ ironic node-list
+--------------------------------------+-----------------+---------------+-------------+--------------------+-------------+
| UUID                                 | Name            | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+-----------------+---------------+-------------+--------------------+-------------+
| 83ba1391-d596-4651-9b58-b1258ebb0652 | overcloud-ceph1 | None          | power off   | available          | False       |
| 379e4486-d9f9-46d5-9e68-b332257c89e0 | overcloud-ceph2 | None          | power off   | available          | False       |
| b9eb9a38-1fe0-4b8f-bd2d-806c8e4a7988 | overcloud-ceph3 | None          | power off   | available          | False       |
+--------------------------------------+-----------------+---------------+-------------+--------------------+-------------+
[stack@undercloud ~]$ openstack baremetal introspection bulk start
Setting nodes for introspection to manageable...
Starting introspection of node: 83ba1391-d596-4651-9b58-b1258ebb0652
Starting introspection of node: 379e4486-d9f9-46d5-9e68-b332257c89e0
Starting introspection of node: b9eb9a38-1fe0-4b8f-bd2d-806c8e4a7988
Waiting for introspection to finish...
Introspection for UUID 379e4486-d9f9-46d5-9e68-b332257c89e0 finished successfully.
~~~

Switching all interfaces to e1000 does not work. All interface with virtio does not work.

[root@rhospbl-5 ~]# virsh list --all | grep ceph | awk '{print $2}' | while read node;do echo -e "\n\n$node";echo "===="; virsh dumpxml $node | grep mac -A2;done


overcloud-ceph1
====
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
    <boot dev='network'/>
  </os>
  <features>
--
      <mac address='52:54:00:dd:93:83'/>
      <source network='provisioning' bridge='virbr2'/>
      <target dev='vnet18'/>
      <model type='virtio'/>
--
      <mac address='52:54:00:e8:bb:79'/>
      <source network='trunk' bridge='virbr1'/>
      <target dev='vnet19'/>
      <model type='virtio'/>
--
      <mac address='52:54:00:44:ea:6f'/>
      <source bridge='br0'/>
      <target dev='vnet20'/>
      <model type='virtio'/>


overcloud-ceph3
====
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
    <boot dev='network'/>
  </os>
  <features>
--
      <mac address='52:54:00:13:97:cb'/>
      <source network='provisioning' bridge='virbr2'/>
      <target dev='vnet24'/>
      <model type='e1000'/>
--
      <mac address='52:54:00:cb:79:5c'/>
      <source network='trunk' bridge='virbr1'/>
      <target dev='vnet25'/>
      <model type='e1000'/>
--
      <mac address='52:54:00:55:be:d1'/>
      <source network='redhat' bridge='br0'/>
      <target dev='vnet26'/>
      <model type='e1000'/>


overcloud-ceph2
====
    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
    <boot dev='network'/>
  </os>
  <features>
--
      <mac address='52:54:00:c4:aa:1a'/>
      <source network='provisioning'/>
      <model type='e1000'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
--
      <mac address='52:54:00:5b:9d:f7'/>
      <source network='trunk'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
--
      <mac address='52:54:00:38:ef:38'/>
      <source network='redhat'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
[root@rhospbl-5 ~]# 


I can reproduce this in any combination (make ceph0, ceph1 or ceph2 work simply by changing network types).

Comment 2 Andreas Karis 2016-04-12 19:18:26 UTC
Created attachment 1146618 [details]
screenshot of fail

Comment 3 Andreas Karis 2016-04-12 19:19:01 UTC
Created attachment 1146619 [details]
e1000-virtio-virtio vs virtio-virtio-virtio

Comment 4 Andreas Karis 2016-04-12 19:19:18 UTC
Created attachment 1146620 [details]
fail

Comment 5 Andreas Karis 2016-04-12 19:19:34 UTC
Created attachment 1146621 [details]
fail

Comment 6 Andreas Karis 2016-04-12 19:19:53 UTC
Created attachment 1146622 [details]
fail

Comment 7 Andreas Karis 2016-04-12 19:20:19 UTC
This seems to resemble: https://bugzilla.redhat.com/show_bug.cgi?id=1209176

Comment 8 Andreas Karis 2016-04-12 19:34:43 UTC
[root@rhospbl-5 ~]# virsh net-dumpxml provisioning
<network connections='3'>
  <name>provisioning</name>
  <uuid>de1d3965-2695-4a1d-9153-ba1e5c268e4e</uuid>
  <bridge name='virbr2' stp='on' delay='0'/>
  <mac address='52:54:00:ad:34:2f'/>
</network>

[root@rhospbl-5 ~]# virsh net-dumpxml trunk
<network connections='4'>
  <name>trunk</name>
  <uuid>3858a7e6-d70f-4842-90f7-2d1b7fee51d9</uuid>
  <bridge name='virbr1' stp='on' delay='0'/>
  <mac address='52:54:00:8b:d4:fb'/>
</network>

[root@rhospbl-5 ~]# virsh net-dumpxml redhat
<network connections='3'>
  <name>redhat</name>
  <uuid>882f0fc6-37d7-4368-a388-9ec5993e0baf</uuid>
  <forward mode='bridge'/>
  <bridge name='br0'/>
</network>

Comment 9 Andreas Karis 2016-04-12 19:45:18 UTC
Switching all interfaces to Nic1: e1000 Nic2: virtio Nic3: virtio makes introspection work

[stack@undercloud ~]$ openstack baremetal introspection bulk start
Setting nodes for introspection to manageable...
Starting introspection of node: 5a484be8-4963-42fe-ba03-b1c228996ce6
Starting introspection of node: af984d74-67ca-4917-b760-5b313994b2a0
Starting introspection of node: 8a36b1dc-a626-414d-b601-cf7a29ba6a7e
Starting introspection of node: 1d541fa2-36d1-43a6-a7f6-af7f1fe09838
Starting introspection of node: a5cdd130-f41c-42ec-a681-4807c8fd318a
Starting introspection of node: b66b8835-993f-49fa-bb22-e93d3d6bc25f
Starting introspection of node: 6652a3b1-3b30-4c99-b934-ed1b09d72b8d
Starting introspection of node: 3b03d4fe-f478-4655-9ef6-f6a861e67c68
Waiting for introspection to finish...
Introspection for UUID 5a484be8-4963-42fe-ba03-b1c228996ce6 finished successfully.
Introspection for UUID 8a36b1dc-a626-414d-b601-cf7a29ba6a7e finished successfully.
Introspection for UUID af984d74-67ca-4917-b760-5b313994b2a0 finished successfully.
Introspection for UUID a5cdd130-f41c-42ec-a681-4807c8fd318a finished successfully.
Introspection for UUID 6652a3b1-3b30-4c99-b934-ed1b09d72b8d finished successfully.
Introspection for UUID b66b8835-993f-49fa-bb22-e93d3d6bc25f finished successfully.
Introspection for UUID 3b03d4fe-f478-4655-9ef6-f6a861e67c68 finished successfully.
Introspection for UUID 1d541fa2-36d1-43a6-a7f6-af7f1fe09838 finished successfully.
Setting manageable nodes to available...
Node 5a484be8-4963-42fe-ba03-b1c228996ce6 has been set to available.
Node af984d74-67ca-4917-b760-5b313994b2a0 has been set to available.
Node 8a36b1dc-a626-414d-b601-cf7a29ba6a7e has been set to available.
Node 1d541fa2-36d1-43a6-a7f6-af7f1fe09838 has been set to available.
Node a5cdd130-f41c-42ec-a681-4807c8fd318a has been set to available.
Node b66b8835-993f-49fa-bb22-e93d3d6bc25f has been set to available.
Node 6652a3b1-3b30-4c99-b934-ed1b09d72b8d has been set to available.
Node 3b03d4fe-f478-4655-9ef6-f6a861e67c68 has been set to available.
Introspection completed.
[stack@undercloud ~]$

Comment 10 Andreas Karis 2016-04-12 19:47:09 UTC
Created attachment 1146641 [details]
xml of all vms

Comment 11 Andreas Karis 2016-04-12 19:47:53 UTC
Created attachment 1146642 [details]
ramdisk logs

Comment 12 Andreas Karis 2016-04-12 19:48:38 UTC
Created attachment 1146643 [details]
ramdisk logs

Comment 13 Andreas Karis 2016-04-12 19:52:01 UTC
Created attachment 1146644 [details]
ramdisk logs

Comment 14 Andreas Karis 2016-04-12 19:52:29 UTC
Created attachment 1146645 [details]
ramdisk logs

Comment 15 Andreas Karis 2016-04-12 19:53:41 UTC
Created attachment 1146646 [details]
ramdisk logs

Comment 16 Andreas Karis 2016-04-12 20:01:21 UTC
Created attachment 1146647 [details]
ram disk

Comment 17 Andreas Karis 2016-04-12 20:01:55 UTC
Created attachment 1146648 [details]
ramdisk logs

Comment 18 Andreas Karis 2016-04-12 20:02:36 UTC
Created attachment 1146649 [details]
ramdisk logs

Comment 19 Andreas Karis 2016-04-12 20:04:04 UTC
Created attachment 1146650 [details]
ramdisk logs

Comment 20 Dmitry Tantsur 2016-08-19 08:17:24 UTC

*** This bug has been marked as a duplicate of bug 1234601 ***


Note You need to log in before you can comment on or make changes to this bug.