Created attachment 1011282 [details] instackenv.json Description of problem: I installed the latest delorean packages (beginning of sprint 5) on a bare metal server and tried to detect the other bare metal nodes I have in my network. Introspection fails and there is nothing in /vat/log/messages to help me find out what the problem is. Version-Release number of selected component (if applicable): python-rdomanager-oscplugin-0.0.1-c2c9653.el7.centos.noarch How reproducible: 100% Steps to Reproduce: 1. instack-ironic-deployment --nodes-json instackenv.json --register-nodes 2. openstack baremetal introspection all start 3. openstack baremetal introspection all status Actual results: Introspection takes a very long time (I left it running over the weekend so I don't know exactly how much) and then times out: openstack baremetal introspection all status /usr/lib/python2.7/site-packages/novaclient/v1_1/__init__.py:30: UserWarning: Module novaclient.v1_1 is deprecated (taken as a basis for novaclient.v2). The preferable way to get client class or object you can find in novaclient.client module. warnings.warn("Module novaclient.v1_1 is deprecated (taken as a basis for " +--------------------------------------+----------+-----------------------+ | Node UUID | Finished | Error | +--------------------------------------+----------+-----------------------+ | 9419b513-3ac6-4b86-a236-81a67f94c1f1 | True | Introspection timeout | | 128db386-50e7-40ac-802e-ea6d3d674e98 | True | Introspection timeout | | d517ecfb-2656-4a82-816a-454e94aa2518 | True | Introspection timeout | | 3806c1b7-eec6-4c8c-b4dc-bbe4ba393255 | True | Introspection timeout | +--------------------------------------+----------+-----------------------+ Additional info: My instackenv.json file is attached.
Dmirty, can you work with Udi on this one to figure out what the issue is?
Dmitry, can you work with Udi on this one to figure out what the issue is?
Hi! Is it possible you connect to the baremetal machines with the vendor remote console and make a screenshot of what is going on there? Also all logs are in journald, please provide output of $ sudo journalctl -u openstack-ironic-discoverd. Thanks.
Created attachment 1011635 [details] journal log Attaching the journal log. I am still looking for a way to open a console to the machines.
Have the same behavior with VMs. $ sudo journalctl -u openstack-ironic-discoverd -- Logs begin at Thu 2015-04-02 09:50:28 EDT, end at Wed 2015-04-08 06:22:40 EDT. -- Apr 07 08:31:58 localhost.localdomain systemd[1]: Stopping Hardware introspection service for OpenStack Ironic... Apr 07 08:31:58 localhost.localdomain systemd[1]: Starting Hardware introspection service for OpenStack Ironic... Apr 07 08:31:58 localhost.localdomain systemd[1]: Started Hardware introspection service for OpenStack Ironic. Apr 07 08:31:59 localhost.localdomain ironic-discoverd[15616]: INFO:werkzeug: * Running on http://0.0.0.0:5050/ Apr 07 08:35:51 localhost.localdomain ironic-discoverd[15616]: ERROR:ironic_discoverd.utils:Could not find node 904e28ad-40a0-45b9-befc-21388f615bc8 in cache Apr 07 08:35:51 localhost.localdomain ironic-discoverd[15616]: INFO:werkzeug:127.0.0.1 - - [07/Apr/2015 08:35:51] "GET /v1/introspection/904e28ad-40a0-45b9-befc-21388f615bc8 HTTP/1.1" 404 - Apr 07 08:35:59 localhost.localdomain ironic-discoverd[15616]: INFO:werkzeug:127.0.0.1 - - [07/Apr/2015 08:35:59] "POST /v1/introspection/904e28ad-40a0-45b9-befc-21388f615bc8 HTTP/1.1" 202 - Apr 07 08:35:59 localhost.localdomain ironic-discoverd[15616]: INFO:ironic_discoverd.introspect:Whitelisting MAC's [u'00:14:1c:9f:93:94'] for node 904e28ad-40a0-45b9-befc-21388f615bc8 on the firewall Apr 07 08:36:00 localhost.localdomain ironic-discoverd[15616]: INFO:werkzeug:127.0.0.1 - - [07/Apr/2015 08:36:00] "POST /v1/introspection/f40a3388-c337-4083-bb5b-ca7b7a877728 HTTP/1.1" 202 - Apr 07 08:36:00 localhost.localdomain ironic-discoverd[15616]: INFO:ironic_discoverd.introspect:Whitelisting MAC's [u'00:ef:d8:a0:58:80'] for node f40a3388-c337-4083-bb5b-ca7b7a877728 on the firewall Apr 07 08:38:10 localhost.localdomain ironic-discoverd[15616]: INFO:werkzeug:127.0.0.1 - - [07/Apr/2015 08:38:10] "GET /v1/introspection/904e28ad-40a0-45b9-befc-21388f615bc8 HTTP/1.1" 200 - Apr 07 08:38:10 localhost.localdomain ironic-discoverd[15616]: INFO:werkzeug:127.0.0.1 - - [07/Apr/2015 08:38:10] "GET /v1/introspection/f40a3388-c337-4083-bb5b-ca7b7a877728 HTTP/1.1" 200 - Apr 07 08:41:34 localhost.localdomain ironic-discoverd[15616]: INFO:werkzeug:127.0.0.1 - - [07/Apr/2015 08:41:34] "GET /v1/introspection/904e28ad-40a0-45b9-befc-21388f615bc8 HTTP/1.1" 200 - Apr 07 08:41:34 localhost.localdomain ironic-discoverd[15616]: INFO:werkzeug:127.0.0.1 - - [07/Apr/2015 08:41:34] "GET /v1/introspection/f40a3388-c337-4083-bb5b-ca7b7a877728 HTTP/1.1" 200 - Apr 07 08:43:05 localhost.localdomain ironic-discoverd[15616]: INFO:werkzeug:127.0.0.1 - - [07/Apr/2015 08:43:05] "GET /v1/introspection/904e28ad-40a0-45b9-befc-21388f615bc8 HTTP/1.1" 200 - Apr 07 08:43:05 localhost.localdomain ironic-discoverd[15616]: INFO:werkzeug:127.0.0.1 - - [07/Apr/2015 08:43:05] "GET /v1/introspection/f40a3388-c337-4083-bb5b-ca7b7a877728 HTTP/1.1" 200 - Apr 07 09:35:59 localhost.localdomain ironic-discoverd[15616]: ERROR:ironic_discoverd.node_cache:Introspection for nodes [u'904e28ad-40a0-45b9-befc-21388f615bc8'] has timed out Apr 07 09:36:59 localhost.localdomain ironic-discoverd[15616]: ERROR:ironic_discoverd.node_cache:Introspection for nodes [u'f40a3388-c337-4083-bb5b-ca7b7a877728'] has timed out Apr 08 06:20:33 localhost.localdomain ironic-discoverd[15616]: INFO:werkzeug:127.0.0.1 - - [08/Apr/2015 06:20:33] "GET /v1/introspection/904e28ad-40a0-45b9-befc-21388f615bc8 HTTP/1.1" 200 - Apr 08 06:20:33 localhost.localdomain ironic-discoverd[15616]: INFO:werkzeug:127.0.0.1 - - [08/Apr/2015 06:20:33] "GET /v1/introspection/f40a3388-c337-4083-bb5b-ca7b7a877728 HTTP/1.1" 200 -
Hi Ola! As you're on VM's it should be pretty easy to make a screenshot using virt-manager. Otherwise we can't figure out if it's the same problem. Also Ola and Udi, please provide $ sudo journalctl -u openstack-ironic-discoverd-dnsmasq
Created attachment 1012264 [details] screenshot
Created attachment 1012265 [details] journalctl
The screenshot is not a screenshot of the machine booting, it's screenshot of the virt-manager itself. You can open a machine and get access to it's virtual screen. However, the log does give some clues. What's in your /tftpboot directory? Did you follow https://repos.fedorapeople.org/repos/openstack-m/instack-undercloud/internal-html/build-images.html before starting discovery?
No vendor remote console for these machines, unfortunately. However our IT guy looked at the machine when it booted and saw the error: "no DHCP offers were received". We are currently investigating if the switch's ports are configured properly for the right vlan, but we need to wait till next week because all the relevant people are on Easter/Passover vacation...
Really an ancient bug, it doesn't happen any more. Closing.