Description of problem:
OSP11 -> OSP12 upgrade: upgrading an environment with the management network activated fails with: resources.ManagementPort: InternalServerError: resources.ManagementPort: Request Failed: internal server error while processing your request.


Checking /var/log/neutron/server.log we can see the following traces:

2017-09-19 12:28:36.606 15057 ERROR neutron.pecan_wsgi.hooks.translation DBError: (pymysql.err.ProgrammingError) (1064, u"You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '} WHERE ml2_port_bindings.port_id = '334d7f0e-8ed6-448a-921b-1b1e043e9efd' AND m' at line 1") [SQL: u'UPDATE ml2_port_bindings SET vnic_type=%(vnic_type)s WHERE ml2_port_bindings.port_id = %(ml2_port_bindings_port_id)s AND ml2_port_bindings.host = %(ml2_port_bindings_host)s'] [parameters: {'vnic_type': {}, 'ml2_port_bindings_host': u'', 'ml2_port_bindings_port_id': u'334d7f0e-8ed6-448a-921b-1b1e043e9efd'}]

334d7f0e-8ed6-448a-921b-1b1e043e9efd is an existing port on the mangement network:

(undercloud) [stack@undercloud-0 ~]$ neutron port-list | grep 334d7f0e-8ed6-448a-921b-1b1e043e9efd
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
| 334d7f0e-8ed6-448a-921b-1b1e043e9efd |                               | f76c192a0a074795b4fe8187a55a5e6f | fa:16:3e:bf:42:18 | {"subnet_id": "e280edd2-6edd-440a-9f67-01b2cd143093", "ip_address": "172.16.17.182"} |

(undercloud) [stack@undercloud-0 ~]$ grep ManagementAllocation ~/openstack_deployment/environments/network-environment.yaml
  ManagementAllocationPools: [{'start': '172.16.17.181', 'end': '172.16.17.210'}]

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-7.0.0-0.20170913050523.0rc2.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP11 with 1 controller + 1 compute with management network enabled:

source ~/stackrc
export THT=/usr/share/openstack-tripleo-heat-templates/

openstack overcloud deploy --templates $THT \
-e $THT/environments/network-isolation.yaml \
-e $THT/environments/network-management.yaml \
-e ~/openstack_deployment/environments/nodes.yaml \
-e ~/openstack_deployment/environments/network-environment.yaml \
-e ~/openstack_deployment/environments/neutron-settings.yaml 

2. Upgrade undercloud to OSP12

3. Make a copy of network_data.yaml from /usr/share/openstack-tripleo-heat-templates/network_data.yaml and set enabled: true for Management network

4. Create a script for running major upgrade composable steps including the custom networks file:

source ~/stackrc
export THT=/usr/share/openstack-tripleo-heat-templates/

openstack overcloud deploy --templates $THT \
-e $THT/environments/network-isolation.yaml \
-e $THT/environments/network-management.yaml \
-e ~/openstack_deployment/environments/nodes.yaml \
-e ~/openstack_deployment/environments/network-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-composable-steps-docker.yaml \
-e /home/stack/docker-osp12.yaml \
-e ~/openstack_deployment/environments/neutron-settings.yaml  \
-e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml \
-n ~/network_data.yaml

5. Run the upgrade script

Actual results:

Upgrade fails with:

 Stack overcloud UPDATE_FAILED 

overcloud.Controller.0.ManagementPort.ManagementPort:
  resource_type: OS::Neutron::Port
  physical_resource_id: 334d7f0e-8ed6-448a-921b-1b1e043e9efd
  status: UPDATE_FAILED
  status_reason: |
    InternalServerError: resources.ManagementPort: Request Failed: internal server error while processing your request.
    Neutron server returns request_ids: ['req-e5973406-e7e3-496f-9909-d40e84fd00dd']
overcloud.Compute.0.ManagementPort.ManagementPort:
  resource_type: OS::Neutron::Port
  physical_resource_id: 0a74d1ce-05a3-447a-a494-27652d2761b7
  status: UPDATE_FAILED
  status_reason: |
    InternalServerError: resources.ManagementPort: Request Failed: internal server error while processing your request.
    Neutron server returns request_ids: ['req-8979467a-e653-4726-9849-ef63011afc2d']
Heat Stack update failed.
Heat Stack update failed.


Expected results:

Upgrade completes fine.

Additional info:

I am adjust the management network during upgrade as discussed in https://bugzilla.redhat.com/show_bug.cgi?id=1488837#c8


[root@undercloud-0 stack]# grep req-e5973406-e7e3-496f-9909-d40e84fd00dd /var/log/neutron/server.log 
2017-09-19 12:28:36.328 15057 WARNING oslo_config.cfg [req-e5973406-e7e3-496f-9909-d40e84fd00dd 0ceb5cb59ea342df9f6c885a65c4f8ed f76c192a0a074795b4fe8187a55a5e6f - default default] Option "rabbit_hosts" from group "oslo_messaging_rabbit" is deprecated for removal (Replaced by [DEFAULT]/transport_url).  Its value may be silently ignored in the future.
2017-09-19 12:28:36.328 15057 WARNING oslo_config.cfg [req-e5973406-e7e3-496f-9909-d40e84fd00dd 0ceb5cb59ea342df9f6c885a65c4f8ed f76c192a0a074795b4fe8187a55a5e6f - default default] Option "rabbit_userid" from group "oslo_messaging_rabbit" is deprecated for removal (Replaced by [DEFAULT]/transport_url).  Its value may be silently ignored in the future.
2017-09-19 12:28:36.329 15057 WARNING oslo_config.cfg [req-e5973406-e7e3-496f-9909-d40e84fd00dd 0ceb5cb59ea342df9f6c885a65c4f8ed f76c192a0a074795b4fe8187a55a5e6f - default default] Option "rabbit_password" from group "oslo_messaging_rabbit" is deprecated for removal (Replaced by [DEFAULT]/transport_url).  Its value may be silently ignored in the future.
2017-09-19 12:28:36.602 15057 ERROR oslo_db.sqlalchemy.exc_filters [req-e5973406-e7e3-496f-9909-d40e84fd00dd 0ceb5cb59ea342df9f6c885a65c4f8ed f76c192a0a074795b4fe8187a55a5e6f - default default] DBAPIError exception wrapped from (pymysql.err.ProgrammingError) (1064, u"You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '} WHERE ml2_port_bindings.port_id = '334d7f0e-8ed6-448a-921b-1b1e043e9efd' AND m' at line 1") [SQL: u'UPDATE ml2_port_bindings SET vnic_type=%(vnic_type)s WHERE ml2_port_bindings.port_id = %(ml2_port_bindings_port_id)s AND ml2_port_bindings.host = %(ml2_port_bindings_host)s'] [parameters: {'vnic_type': {}, 'ml2_port_bindings_host': u'', 'ml2_port_bindings_port_id': u'334d7f0e-8ed6-448a-921b-1b1e043e9efd'}]: ProgrammingError: (1064, u"You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '} WHERE ml2_port_bindings.port_id = '334d7f0e-8ed6-448a-921b-1b1e043e9efd' AND m' at line 1")
2017-09-19 12:28:36.606 15057 ERROR neutron.pecan_wsgi.hooks.translation [req-e5973406-e7e3-496f-9909-d40e84fd00dd 0ceb5cb59ea342df9f6c885a65c4f8ed f76c192a0a074795b4fe8187a55a5e6f - default default] PUT failed.: DBError: (pymysql.err.ProgrammingError) (1064, u"You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '} WHERE ml2_port_bindings.port_id = '334d7f0e-8ed6-448a-921b-1b1e043e9efd' AND m' at line 1") [SQL: u'UPDATE ml2_port_bindings SET vnic_type=%(vnic_type)s WHERE ml2_port_bindings.port_id = %(ml2_port_bindings_port_id)s AND ml2_port_bindings.host = %(ml2_port_bindings_host)s'] [parameters: {'vnic_type': {}, 'ml2_port_bindings_host': u'', 'ml2_port_bindings_port_id': u'334d7f0e-8ed6-448a-921b-1b1e043e9efd'}]
2017-09-19 12:28:36.612 15057 INFO neutron.wsgi [req-e5973406-e7e3-496f-9909-d40e84fd00dd 0ceb5cb59ea342df9f6c885a65c4f8ed f76c192a0a074795b4fe8187a55a5e6f - default default] 192.168.0.1 "PUT /v2.0/ports/334d7f0e-8ed6-448a-921b-1b1e043e9efd HTTP/1.1" status: 500  len: 368 time: 0.2926481
[root@undercloud-0 stack]# 
[root@undercloud-0 stack]# 
[root@undercloud-0 stack]# grep req-8979467a-e653-4726-9849-ef63011afc2d /var/log/neutron/server.log 
2017-09-19 12:28:30.029 15056 WARNING oslo_config.cfg [req-8979467a-e653-4726-9849-ef63011afc2d 0ceb5cb59ea342df9f6c885a65c4f8ed f76c192a0a074795b4fe8187a55a5e6f - default default] Option "rabbit_hosts" from group "oslo_messaging_rabbit" is deprecated for removal (Replaced by [DEFAULT]/transport_url).  Its value may be silently ignored in the future.
2017-09-19 12:28:30.030 15056 WARNING oslo_config.cfg [req-8979467a-e653-4726-9849-ef63011afc2d 0ceb5cb59ea342df9f6c885a65c4f8ed f76c192a0a074795b4fe8187a55a5e6f - default default] Option "rabbit_userid" from group "oslo_messaging_rabbit" is deprecated for removal (Replaced by [DEFAULT]/transport_url).  Its value may be silently ignored in the future.
2017-09-19 12:28:30.030 15056 WARNING oslo_config.cfg [req-8979467a-e653-4726-9849-ef63011afc2d 0ceb5cb59ea342df9f6c885a65c4f8ed f76c192a0a074795b4fe8187a55a5e6f - default default] Option "rabbit_password" from group "oslo_messaging_rabbit" is deprecated for removal (Replaced by [DEFAULT]/transport_url).  Its value may be silently ignored in the future.
2017-09-19 12:28:30.453 15056 ERROR oslo_db.sqlalchemy.exc_filters [req-8979467a-e653-4726-9849-ef63011afc2d 0ceb5cb59ea342df9f6c885a65c4f8ed f76c192a0a074795b4fe8187a55a5e6f - default default] DBAPIError exception wrapped from (pymysql.err.ProgrammingError) (1064, u"You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '} WHERE ml2_port_bindings.port_id = '0a74d1ce-05a3-447a-a494-27652d2761b7' AND m' at line 1") [SQL: u'UPDATE ml2_port_bindings SET vnic_type=%(vnic_type)s WHERE ml2_port_bindings.port_id = %(ml2_port_bindings_port_id)s AND ml2_port_bindings.host = %(ml2_port_bindings_host)s'] [parameters: {'vnic_type': {}, 'ml2_port_bindings_host': u'', 'ml2_port_bindings_port_id': u'0a74d1ce-05a3-447a-a494-27652d2761b7'}]: ProgrammingError: (1064, u"You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '} WHERE ml2_port_bindings.port_id = '0a74d1ce-05a3-447a-a494-27652d2761b7' AND m' at line 1")
2017-09-19 12:28:30.458 15056 ERROR neutron.pecan_wsgi.hooks.translation [req-8979467a-e653-4726-9849-ef63011afc2d 0ceb5cb59ea342df9f6c885a65c4f8ed f76c192a0a074795b4fe8187a55a5e6f - default default] PUT failed.: DBError: (pymysql.err.ProgrammingError) (1064, u"You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '} WHERE ml2_port_bindings.port_id = '0a74d1ce-05a3-447a-a494-27652d2761b7' AND m' at line 1") [SQL: u'UPDATE ml2_port_bindings SET vnic_type=%(vnic_type)s WHERE ml2_port_bindings.port_id = %(ml2_port_bindings_port_id)s AND ml2_port_bindings.host = %(ml2_port_bindings_host)s'] [parameters: {'vnic_type': {}, 'ml2_port_bindings_host': u'', 'ml2_port_bindings_port_id': u'0a74d1ce-05a3-447a-a494-27652d2761b7'}]
2017-09-19 12:28:30.465 15056 INFO neutron.wsgi [req-8979467a-e653-4726-9849-ef63011afc2d 0ceb5cb59ea342df9f6c885a65c4f8ed f76c192a0a074795b4fe8187a55a5e6f - default default] 192.168.0.1 "PUT /v2.0/ports/0a74d1ce-05a3-447a-a494-27652d2761b7 HTTP/1.1" status: 500  len: 368 time: 0.4439561
[root@undercloud-0 stack]# 

Attaching the sosreport from undercloud.