Bug 1572165 - Fails to start nodes as MAC address already in use with ODL deployments
Summary: Fails to start nodes as MAC address already in use with ODL deployments
Keywords:
Status: CLOSED DUPLICATE of bug 1571384
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: puppet-ironic
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: RHOS Maint
QA Contact: nlevinki
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-26 09:59 UTC by Tomas Jamrisko
Modified: 2018-04-26 20:08 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-04-26 13:38:25 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
Logs from the undercloud (17.27 MB, application/x-gzip)
2018-04-26 09:59 UTC, Tomas Jamrisko 		no flags Details
View All

Description Tomas Jamrisko 2018-04-26 09:59:02 UTC 
Created attachment 1427118 [details]
Logs from the undercloud

Description of problem:
Randomly hitting issues where one or multiple nodes fail to start during deployment of overcloud


How reproducible:
Random

Steps to Reproduce:
1. Try to get an openstack deployment in a virtualized setup

Actual results:
One or more nodes end in either ERROR or BUILD state


Additional info:

./undercloud-0/var/log/neutron/server.log:8063:2018-04-25 12:12:54.435 18323 DEBUG neutron.api.v2.base [req-4efbbb2e-5dab-401e-ab8e-df982868b96d 6caf62ee9a0b4e29810a4e202e464f11 0e191d56e18149fd9a3db6c1cdaf98ae - default default] Request body: {u'port': {u'mac_address': u'52:54:00:7c:c4:56'}} prepare_request_body /usr/lib/python2.7/site-packages/neutron/api/v2/base.py:690
./undercloud-0/var/log/neutron/server.log:8065:2018-04-25 12:12:54.657 18323 DEBUG neutron_lib.callbacks.manager [req-4efbbb2e-5dab-401e-ab8e-df982868b96d 6caf62ee9a0b4e29810a4e202e464f11 0e191d56e18149fd9a3db6c1cdaf98ae - default default] Notify callbacks ['neutron.plugins.ml2.plugin.Ml2Plugin._ensure_default_security_group_handler--9223372036853807880'] for port, before_update _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167
./undercloud-0/var/log/neutron/server.log:8068:2018-04-25 12:12:54.812 18323 INFO neutron.pecan_wsgi.hooks.translation [req-4efbbb2e-5dab-401e-ab8e-df982868b96d 6caf62ee9a0b4e29810a4e202e464f11 0e191d56e18149fd9a3db6c1cdaf98ae - default default] PUT failed (client error): There was a conflict when trying to complete your request.
./undercloud-0/var/log/neutron/server.log:8069:2018-04-25 12:12:54.813 18323 DEBUG neutron.pecan_wsgi.hooks.notifier [req-4efbbb2e-5dab-401e-ab8e-df982868b96d 6caf62ee9a0b4e29810a4e202e464f11 0e191d56e18149fd9a3db6c1cdaf98ae - default default] No notification will be sent due to unsuccessful status code: 409 after /usr/lib/python2.7/site-packages/neutron/pecan_wsgi/hooks/notifier.py:79
./undercloud-0/var/log/neutron/server.log:8070:2018-04-25 12:12:54.814 18323 INFO neutron.wsgi [req-4efbbb2e-5dab-401e-ab8e-df982868b96d 6caf62ee9a0b4e29810a4e202e464f11 0e191d56e18149fd9a3db6c1cdaf98ae - default default] 192.168.24.1 "PUT /v2.0/ports/305beda7-0dd0-4fec-b4d2-78b7aa4e8e6a HTTP/1.1" status: 409  len: 378 time: 0.3846960
./undercloud-0/var/log/ironic/ironic-conductor.log:4931:Neutron server returns request_ids: ['req-4efbbb2e-5dab-401e-ab8e-df982868b96d']
./undercloud-0/var/log/ironic/ironic-conductor.log:4950:2018-04-25 12:12:54.817 16411 ERROR ironic.common.neutron Neutron server returns request_ids: ['req-4efbbb2e-5dab-401e-ab8e-df982868b96d']
./undercloud-0/var/log/messages:54864:Apr 25 12:12:54 undercloud-0 ironic-conductor: Neutron server returns request_ids: ['req-4efbbb2e-5dab-401e-ab8e-df982868b96d']
./undercloud-0/var/log/messages:54883:Apr 25 12:12:54 undercloud-0 ironic-conductor: 2018-04-25 12:12:54.817 16411 ERROR ironic.common.neutron Neutron server returns request_ids: ['req-4efbbb2e-5dab-401e-ab8e-df982868b96d']
Comment 1 Bob Fournier 2018-04-26 13:38:25 UTC 
For the logs we see the "mac address xxxx is in use" issue:
var/log/messages:Apr 25 12:12:54 undercloud-0 ironic-conductor: 2018-04-25 12:12:54.817 16411 ERROR ironic.common.neutron [req-97622d5d-cbd1-4993-aca9-53bce291c4e5 6caf62ee9a0b4e29810a4e202e464f11 0e191d56e18149fd9a3db6c1cdaf98ae - default default] Failed to update MAC address on Neutron port 305beda7-0dd0-4fec-b4d2-78b7aa4e8e6a.: MacAddressInUseClient: Unable to complete operation for network 1e252627-6223-4076-a2b9-6f56493c9bac. The mac address 52:54:00:7c:c4:56 is in use.

and:
var/log/ironic/ironic-conductor.log:2018-04-25 12:12:54.817 16411 ERROR ironic.common.neutron [req-97622d5d-cbd1-4993-aca9-53bce291c4e5 6caf62ee9a0b4e29810a4e202e464f11 0e191d56e18149fd9a3db6c1cdaf98ae - default default] Failed to update MAC address on Neutron port 305beda7-0dd0-4fec-b4d2-78b7aa4e8e6a.: MacAddressInUseClient: Unable to complete operation for network 1e252627-6223-4076-a2b9-6f56493c9bac. The mac address 52:54:00:7c:c4:56 is in use.


This is being tracked in https://bugzilla.redhat.com/show_bug.cgi?id=1571384, so marking this one as a duplicate.

*** This bug has been marked as a duplicate of bug 1571384 ***
Comment 2 Bob Fournier 2018-04-26 20:08:30 UTC 
Also note in the logs that there were ipmitool power on failures, we're thinking that is related to source of problem and will follow up in https://bugzilla.redhat.com/show_bug.cgi?id=1571384

ironic/ironic-conductor.log:2018-04-25 12:12:13.327 16411 ERROR ironic.drivers.modules.ipmitool [req-b464e608-071d-4b77-818b-3fb9a6d1369f - - - - -] IPMI Error while attempting "ipmitool -I lanplus -H 172.16.0.1 -L ADMINISTRATOR -p 6234 -U admin -R 12 -N 5 -f /tmp/tmpyUhy5C power on" for node 82fe7d85-7040-4fa0-bb7a-c2127b58415b. Error: Unexpected error while running command.
ironic/ironic-conductor.log:2018-04-25 12:12:13.328 16411 WARNING ironic.drivers.modules.ipmitool [req-b464e608-071d-4b77-818b-3fb9a6d1369f - - - - -] IPMI power action power on failed for node 82fe7d85-7040-4fa0-bb7a-c2127b58415b with error: Unexpected error while running command.
ironic/ironic-conductor.log:2018-04-25 12:12:13.332 16411 ERROR ironic.drivers.modules.ipmitool [req-826d0904-ec2f-473f-8057-7099db9f094d - - - - -] IPMI Error while attempting "ipmitool -I lanplus -H 172.16.0.1 -L ADMINISTRATOR -p 6232 -U admin -R 12 -N 5 -f /tmp/tmpO_RujP power on" for node 4091dfad-9575-49cd-8c0a-4661cf62674f. Error: Unexpected error while running command.
ironic/ironic-conductor.log:2018-04-25 12:12:13.332 16411 WARNING ironic.drivers.modules.ipmitool [req-826d0904-ec2f-473f-8057-7099db9f094d - - - - -] IPMI power action power on failed for node 4091dfad-9575-49cd-8c0a-4661cf62674f with error: Unexpected error while running command.
ironic/ironic-conductor.log:2018-04-25 12:12:13.364 16411 ERROR ironic.drivers.modules.ipmitool [req-839bd58d-f52c-4c96-b911-1223f741b49a - - - - -] IPMI Error while attempting "ipmitool -I lanplus -H 172.16.0.1 -L ADMINISTRATOR -p 6233 -U admin -R 12 -N 5 -f /tmp/tmp8AMcgp power on" for node aa83332d-31c8-4194-8671-8cdf0b00a2de. Error: Unexpected error while running command.
ironic/ironic-conductor.log:2018-04-25 12:12:13.364 16411 WARNING ironic.drivers.modules.ipmitool [req-839bd58d-f52c-4c96-b911-1223f741b49a - - - - -] IPMI power action power on failed for node aa83332d-31c8-4194-8671-8cdf0b00a2de with error: Unexpected error while running command.
ironic/ironic-conductor.log:2018-04-25 12:12:47.528 16411 ERROR ironic.drivers.modules.ipmitool [req-ea738901-e978-43f8-b3e7-378c41dc888f - - - - -] IPMI Error while attempting "ipmitool -I lanplus -H 172.16.0.1 -L ADMINISTRATOR -p 6231 -U admin -R 12 -N 5 -f /tmp/tmpgCde4F power on" for node 7573fc6e-fadc-44e8-9574-cce49f8eaebd. Error: Unexpected error while running command.
ironic/ironic-conductor.log:2018-04-25 12:12:47.529 16411 WARNING ironic.drivers.modules.ipmitool [req-ea738901-e978-43f8-b3e7-378c41dc888f - - - - -] IPMI power action power on failed for node 7573fc6e-fadc-44e8-9574-cce49f8eaebd with error: Unexpected error while running command.
ironic/ironic-conductor.log:2018-04-25 12:12:51.635 16411 ERROR ironic.drivers.modules.ipmitool [req-fd693f2b-0b74-4718-b736-c7071c7506c4 - - - - -] IPMI Error while attempting "ipmitool -I lanplus -H 172.16.0.1 -L ADMINISTRATOR -p 6230 -U admin -R 12 -N 5 -f /tmp/tmpYgRw11 power on" for node 7d4ba533-9b4a-4db8-8275-1f004a895d82. Error: Unexpected error while running command.
ironic/ironic-conductor.log:2018-04-25 12:12:51.636 16411 WARNING ironic.drivers.modules.ipmitool [req-fd693f2b-0b74-4718-b736-c7071c7506c4 - - - - -] IPMI power action power on failed for node 7d4ba533-9b4a-4db8-8275-1f004a895d82 with error: Unexpected error while running command.
messages:Apr 25 12:12:13 undercloud-0 ironic-conductor: 2018-04-25 12:12:13.327 16411 ERROR ironic.drivers.modules.ipmitool [req-b464e608-071d-4b77-818b-3fb9a6d1369f - - - - -] IPMI Error while attempting "ipmitool -I lanplus -H 172.16.0.1 -L ADMINISTRATOR -p 6234 -U admin -R 12 -N 5 -f /tmp/tmpyUhy5C power on" for node 82fe7d85-7040-4fa0-bb7a-c2127b58415b. Error: Unexpected error while running command.
messages:Apr 25 12:12:13 undercloud-0 ironic-conductor: 2018-04-25 12:12:13.328 16411 WARNING ironic.drivers.modules.ipmitool [req-b464e608-071d-4b77-818b-3fb9a6d1369f - - - - -] IPMI power action power on failed for node 82fe7d85-7040-4fa0-bb7a-c2127b58415b with error: Unexpected error while running command.
messages:Apr 25 12:12:13 undercloud-0 ironic-conductor: 2018-04-25 12:12:13.332 16411 ERROR ironic.drivers.modules.ipmitool [req-826d0904-ec2f-473f-8057-7099db9f094d - - - - -] IPMI Error while attempting "ipmitool -I lanplus -H 172.16.0.1 -L ADMINISTRATOR -p 6232 -U admin -R 12 -N 5 -f /tmp/tmpO_RujP power on" for node 4091dfad-9575-49cd-8c0a-4661cf62674f. Error: Unexpected error while running command.
messages:Apr 25 12:12:13 undercloud-0 ironic-conductor: 2018-04-25 12:12:13.332 16411 WARNING ironic.drivers.modules.ipmitool [req-826d0904-ec2f-473f-8057-7099db9f094d - - - - -] IPMI power action power on failed for node 4091dfad-9575-49cd-8c0a-4661cf62674f with error: Unexpected error while running command.
messages:Apr 25 12:12:13 undercloud-0 ironic-conductor: 2018-04-25 12:12:13.364 16411 ERROR ironic.drivers.modules.ipmitool [req-839bd58d-f52c-4c96-b911-1223f741b49a - - - - -] IPMI Error while attempting "ipmitool -I lanplus -H 172.16.0.1 -L ADMINISTRATOR -p 6233 -U admin -R 12 -N 5 -f /tmp/tmp8AMcgp power on" for node aa83332d-31c8-4194-8671-8cdf0b00a2de. Error: Unexpected error while running command.
messages:Apr 25 12:12:13 undercloud-0 ironic-conductor: 2018-04-25 12:12:13.364 16411 WARNING ironic.drivers.modules.ipmitool [req-839bd58d-f52c-4c96-b911-1223f741b49a - - - - -] IPMI power action power on failed for node aa83332d-31c8-4194-8671-8cdf0b00a2de with error: Unexpected error while running command.
messages:Apr 25 12:12:47 undercloud-0 ironic-conductor: 2018-04-25 12:12:47.528 16411 ERROR ironic.drivers.modules.ipmitool [req-ea738901-e978-43f8-b3e7-378c41dc888f - - - - -] IPMI Error while attempting "ipmitool -I lanplus -H 172.16.0.1 -L ADMINISTRATOR -p 6231 -U admin -R 12 -N 5 -f /tmp/tmpgCde4F power on" for node 7573fc6e-fadc-44e8-9574-cce49f8eaebd. Error: Unexpected error while running command.
messages:Apr 25 12:12:47 undercloud-0 ironic-conductor: 2018-04-25 12:12:47.529 16411 WARNING ironic.drivers.modules.ipmitool [req-ea738901-e978-43f8-b3e7-378c41dc888f - - - - -] IPMI power action power on failed for node 7573fc6e-fadc-44e8-9574-cce49f8eaebd with error: Unexpected error while running command.
messages:Apr 25 12:12:51 undercloud-0 ironic-conductor: 2018-04-25 12:12:51.635 16411 ERROR ironic.drivers.modules.ipmitool [req-fd693f2b-0b74-4718-b736-c7071c7506c4 - - - - -] IPMI Error while attempting "ipmitool -I lanplus -H 172.16.0.1 -L ADMINISTRATOR -p 6230 -U admin -R 12 -N 5 -f /tmp/tmpYgRw11 power on" for node 7d4ba533-9b4a-4db8-8275-1f004a895d82. Error: Unexpected error while running command.
messages:Apr 25 12:12:51 undercloud-0 ironic-conductor: 2018-04-25 12:12:51.636 16411 WARNING ironic.drivers.modules.ipmitool [req-fd693f2b-0b74-4718-b736-c7071c7506c4 - - - - -] IPMI power action power on failed for node 7d4ba533-9b4a-4db8-8275-1f004a895d82 with error: Unexpected error while running command.
Note You need to log in before you can comment on or make changes to this bug.