Bug 1493234 - OSP11 -> OSP12 upgrade: upgrading an environment with the management network activated fails with: resources.ManagementPort: InternalServerError: resources.ManagementPort: Request Failed: internal server error while processing your request.
Summary: OSP11 -> OSP12 upgrade: upgrading an environment with the management network ...
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director
Version: 12.0 (Pike)
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: ga
: 12.0 (Pike)
Assignee: Ihar Hrachyshka
QA Contact: Amit Ugol
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-09-19 16:42 UTC by Marius Cornea
Modified: 2017-10-05 11:45 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-10-04 20:33:13 UTC
Target Upstream Version:


Attachments (Terms of Use)
sosreport (14.95 MB, application/x-xz)
2017-09-19 16:48 UTC, Marius Cornea
no flags Details

Description Marius Cornea 2017-09-19 16:42:27 UTC
Description of problem:
OSP11 -> OSP12 upgrade: upgrading an environment with the management network activated fails with: resources.ManagementPort: InternalServerError: resources.ManagementPort: Request Failed: internal server error while processing your request.


Checking /var/log/neutron/server.log we can see the following traces:

2017-09-19 12:28:36.606 15057 ERROR neutron.pecan_wsgi.hooks.translation DBError: (pymysql.err.ProgrammingError) (1064, u"You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '} WHERE ml2_port_bindings.port_id = '334d7f0e-8ed6-448a-921b-1b1e043e9efd' AND m' at line 1") [SQL: u'UPDATE ml2_port_bindings SET vnic_type=%(vnic_type)s WHERE ml2_port_bindings.port_id = %(ml2_port_bindings_port_id)s AND ml2_port_bindings.host = %(ml2_port_bindings_host)s'] [parameters: {'vnic_type': {}, 'ml2_port_bindings_host': u'', 'ml2_port_bindings_port_id': u'334d7f0e-8ed6-448a-921b-1b1e043e9efd'}]

334d7f0e-8ed6-448a-921b-1b1e043e9efd is an existing port on the mangement network:

(undercloud) [stack@undercloud-0 ~]$ neutron port-list | grep 334d7f0e-8ed6-448a-921b-1b1e043e9efd
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
| 334d7f0e-8ed6-448a-921b-1b1e043e9efd |                               | f76c192a0a074795b4fe8187a55a5e6f | fa:16:3e:bf:42:18 | {"subnet_id": "e280edd2-6edd-440a-9f67-01b2cd143093", "ip_address": "172.16.17.182"} |

(undercloud) [stack@undercloud-0 ~]$ grep ManagementAllocation ~/openstack_deployment/environments/network-environment.yaml
  ManagementAllocationPools: [{'start': '172.16.17.181', 'end': '172.16.17.210'}]

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-7.0.0-0.20170913050523.0rc2.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP11 with 1 controller + 1 compute with management network enabled:

source ~/stackrc
export THT=/usr/share/openstack-tripleo-heat-templates/

openstack overcloud deploy --templates $THT \
-e $THT/environments/network-isolation.yaml \
-e $THT/environments/network-management.yaml \
-e ~/openstack_deployment/environments/nodes.yaml \
-e ~/openstack_deployment/environments/network-environment.yaml \
-e ~/openstack_deployment/environments/neutron-settings.yaml 

2. Upgrade undercloud to OSP12

3. Make a copy of network_data.yaml from /usr/share/openstack-tripleo-heat-templates/network_data.yaml and set enabled: true for Management network

4. Create a script for running major upgrade composable steps including the custom networks file:

source ~/stackrc
export THT=/usr/share/openstack-tripleo-heat-templates/

openstack overcloud deploy --templates $THT \
-e $THT/environments/network-isolation.yaml \
-e $THT/environments/network-management.yaml \
-e ~/openstack_deployment/environments/nodes.yaml \
-e ~/openstack_deployment/environments/network-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-composable-steps-docker.yaml \
-e /home/stack/docker-osp12.yaml \
-e ~/openstack_deployment/environments/neutron-settings.yaml  \
-e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml \
-n ~/network_data.yaml

5. Run the upgrade script

Actual results:

Upgrade fails with:

 Stack overcloud UPDATE_FAILED 

overcloud.Controller.0.ManagementPort.ManagementPort:
  resource_type: OS::Neutron::Port
  physical_resource_id: 334d7f0e-8ed6-448a-921b-1b1e043e9efd
  status: UPDATE_FAILED
  status_reason: |
    InternalServerError: resources.ManagementPort: Request Failed: internal server error while processing your request.
    Neutron server returns request_ids: ['req-e5973406-e7e3-496f-9909-d40e84fd00dd']
overcloud.Compute.0.ManagementPort.ManagementPort:
  resource_type: OS::Neutron::Port
  physical_resource_id: 0a74d1ce-05a3-447a-a494-27652d2761b7
  status: UPDATE_FAILED
  status_reason: |
    InternalServerError: resources.ManagementPort: Request Failed: internal server error while processing your request.
    Neutron server returns request_ids: ['req-8979467a-e653-4726-9849-ef63011afc2d']
Heat Stack update failed.
Heat Stack update failed.


Expected results:

Upgrade completes fine.

Additional info:

I am adjust the management network during upgrade as discussed in https://bugzilla.redhat.com/show_bug.cgi?id=1488837#c8


[root@undercloud-0 stack]# grep req-e5973406-e7e3-496f-9909-d40e84fd00dd /var/log/neutron/server.log 
2017-09-19 12:28:36.328 15057 WARNING oslo_config.cfg [req-e5973406-e7e3-496f-9909-d40e84fd00dd 0ceb5cb59ea342df9f6c885a65c4f8ed f76c192a0a074795b4fe8187a55a5e6f - default default] Option "rabbit_hosts" from group "oslo_messaging_rabbit" is deprecated for removal (Replaced by [DEFAULT]/transport_url).  Its value may be silently ignored in the future.
2017-09-19 12:28:36.328 15057 WARNING oslo_config.cfg [req-e5973406-e7e3-496f-9909-d40e84fd00dd 0ceb5cb59ea342df9f6c885a65c4f8ed f76c192a0a074795b4fe8187a55a5e6f - default default] Option "rabbit_userid" from group "oslo_messaging_rabbit" is deprecated for removal (Replaced by [DEFAULT]/transport_url).  Its value may be silently ignored in the future.
2017-09-19 12:28:36.329 15057 WARNING oslo_config.cfg [req-e5973406-e7e3-496f-9909-d40e84fd00dd 0ceb5cb59ea342df9f6c885a65c4f8ed f76c192a0a074795b4fe8187a55a5e6f - default default] Option "rabbit_password" from group "oslo_messaging_rabbit" is deprecated for removal (Replaced by [DEFAULT]/transport_url).  Its value may be silently ignored in the future.
2017-09-19 12:28:36.602 15057 ERROR oslo_db.sqlalchemy.exc_filters [req-e5973406-e7e3-496f-9909-d40e84fd00dd 0ceb5cb59ea342df9f6c885a65c4f8ed f76c192a0a074795b4fe8187a55a5e6f - default default] DBAPIError exception wrapped from (pymysql.err.ProgrammingError) (1064, u"You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '} WHERE ml2_port_bindings.port_id = '334d7f0e-8ed6-448a-921b-1b1e043e9efd' AND m' at line 1") [SQL: u'UPDATE ml2_port_bindings SET vnic_type=%(vnic_type)s WHERE ml2_port_bindings.port_id = %(ml2_port_bindings_port_id)s AND ml2_port_bindings.host = %(ml2_port_bindings_host)s'] [parameters: {'vnic_type': {}, 'ml2_port_bindings_host': u'', 'ml2_port_bindings_port_id': u'334d7f0e-8ed6-448a-921b-1b1e043e9efd'}]: ProgrammingError: (1064, u"You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '} WHERE ml2_port_bindings.port_id = '334d7f0e-8ed6-448a-921b-1b1e043e9efd' AND m' at line 1")
2017-09-19 12:28:36.606 15057 ERROR neutron.pecan_wsgi.hooks.translation [req-e5973406-e7e3-496f-9909-d40e84fd00dd 0ceb5cb59ea342df9f6c885a65c4f8ed f76c192a0a074795b4fe8187a55a5e6f - default default] PUT failed.: DBError: (pymysql.err.ProgrammingError) (1064, u"You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '} WHERE ml2_port_bindings.port_id = '334d7f0e-8ed6-448a-921b-1b1e043e9efd' AND m' at line 1") [SQL: u'UPDATE ml2_port_bindings SET vnic_type=%(vnic_type)s WHERE ml2_port_bindings.port_id = %(ml2_port_bindings_port_id)s AND ml2_port_bindings.host = %(ml2_port_bindings_host)s'] [parameters: {'vnic_type': {}, 'ml2_port_bindings_host': u'', 'ml2_port_bindings_port_id': u'334d7f0e-8ed6-448a-921b-1b1e043e9efd'}]
2017-09-19 12:28:36.612 15057 INFO neutron.wsgi [req-e5973406-e7e3-496f-9909-d40e84fd00dd 0ceb5cb59ea342df9f6c885a65c4f8ed f76c192a0a074795b4fe8187a55a5e6f - default default] 192.168.0.1 "PUT /v2.0/ports/334d7f0e-8ed6-448a-921b-1b1e043e9efd HTTP/1.1" status: 500  len: 368 time: 0.2926481
[root@undercloud-0 stack]# 
[root@undercloud-0 stack]# 
[root@undercloud-0 stack]# grep req-8979467a-e653-4726-9849-ef63011afc2d /var/log/neutron/server.log 
2017-09-19 12:28:30.029 15056 WARNING oslo_config.cfg [req-8979467a-e653-4726-9849-ef63011afc2d 0ceb5cb59ea342df9f6c885a65c4f8ed f76c192a0a074795b4fe8187a55a5e6f - default default] Option "rabbit_hosts" from group "oslo_messaging_rabbit" is deprecated for removal (Replaced by [DEFAULT]/transport_url).  Its value may be silently ignored in the future.
2017-09-19 12:28:30.030 15056 WARNING oslo_config.cfg [req-8979467a-e653-4726-9849-ef63011afc2d 0ceb5cb59ea342df9f6c885a65c4f8ed f76c192a0a074795b4fe8187a55a5e6f - default default] Option "rabbit_userid" from group "oslo_messaging_rabbit" is deprecated for removal (Replaced by [DEFAULT]/transport_url).  Its value may be silently ignored in the future.
2017-09-19 12:28:30.030 15056 WARNING oslo_config.cfg [req-8979467a-e653-4726-9849-ef63011afc2d 0ceb5cb59ea342df9f6c885a65c4f8ed f76c192a0a074795b4fe8187a55a5e6f - default default] Option "rabbit_password" from group "oslo_messaging_rabbit" is deprecated for removal (Replaced by [DEFAULT]/transport_url).  Its value may be silently ignored in the future.
2017-09-19 12:28:30.453 15056 ERROR oslo_db.sqlalchemy.exc_filters [req-8979467a-e653-4726-9849-ef63011afc2d 0ceb5cb59ea342df9f6c885a65c4f8ed f76c192a0a074795b4fe8187a55a5e6f - default default] DBAPIError exception wrapped from (pymysql.err.ProgrammingError) (1064, u"You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '} WHERE ml2_port_bindings.port_id = '0a74d1ce-05a3-447a-a494-27652d2761b7' AND m' at line 1") [SQL: u'UPDATE ml2_port_bindings SET vnic_type=%(vnic_type)s WHERE ml2_port_bindings.port_id = %(ml2_port_bindings_port_id)s AND ml2_port_bindings.host = %(ml2_port_bindings_host)s'] [parameters: {'vnic_type': {}, 'ml2_port_bindings_host': u'', 'ml2_port_bindings_port_id': u'0a74d1ce-05a3-447a-a494-27652d2761b7'}]: ProgrammingError: (1064, u"You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '} WHERE ml2_port_bindings.port_id = '0a74d1ce-05a3-447a-a494-27652d2761b7' AND m' at line 1")
2017-09-19 12:28:30.458 15056 ERROR neutron.pecan_wsgi.hooks.translation [req-8979467a-e653-4726-9849-ef63011afc2d 0ceb5cb59ea342df9f6c885a65c4f8ed f76c192a0a074795b4fe8187a55a5e6f - default default] PUT failed.: DBError: (pymysql.err.ProgrammingError) (1064, u"You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '} WHERE ml2_port_bindings.port_id = '0a74d1ce-05a3-447a-a494-27652d2761b7' AND m' at line 1") [SQL: u'UPDATE ml2_port_bindings SET vnic_type=%(vnic_type)s WHERE ml2_port_bindings.port_id = %(ml2_port_bindings_port_id)s AND ml2_port_bindings.host = %(ml2_port_bindings_host)s'] [parameters: {'vnic_type': {}, 'ml2_port_bindings_host': u'', 'ml2_port_bindings_port_id': u'0a74d1ce-05a3-447a-a494-27652d2761b7'}]
2017-09-19 12:28:30.465 15056 INFO neutron.wsgi [req-8979467a-e653-4726-9849-ef63011afc2d 0ceb5cb59ea342df9f6c885a65c4f8ed f76c192a0a074795b4fe8187a55a5e6f - default default] 192.168.0.1 "PUT /v2.0/ports/0a74d1ce-05a3-447a-a494-27652d2761b7 HTTP/1.1" status: 500  len: 368 time: 0.4439561
[root@undercloud-0 stack]# 

Attaching the sosreport from undercloud.

Comment 1 Marius Cornea 2017-09-19 16:48:12 UTC
Created attachment 1328065 [details]
sosreport

Comment 5 Assaf Muller 2017-10-02 13:36:56 UTC
Assigned to Ihar for further triage, Jakub is looking at another blocker.

Comment 6 Jakub Libosvar 2017-10-04 11:33:16 UTC
I briefly looked at the sosreport and server logs are missing DEBUG messages. I think it would help if we see the requests coming to Neutron so we're able to create some minimal reproducer for the 500 internal error server issue.

Comment 7 Marius Cornea 2017-10-04 11:34:50 UTC
(In reply to Jakub Libosvar from comment #6)
> I briefly looked at the sosreport and server logs are missing DEBUG
> messages. I think it would help if we see the requests coming to Neutron so
> we're able to create some minimal reproducer for the 500 internal error
> server issue.

I'll get back with the debug logs asap.

Comment 8 Marius Cornea 2017-10-04 20:33:13 UTC
(In reply to Marius Cornea from comment #7)
> (In reply to Jakub Libosvar from comment #6)
> > I briefly looked at the sosreport and server logs are missing DEBUG
> > messages. I think it would help if we see the requests coming to Neutron so
> > we're able to create some minimal reproducer for the 500 internal error
> > server issue.
> 
> I'll get back with the debug logs asap.

I wasn't able to reproduce this issue with the latest puddle build. The reproducing steps(no need to pass -n ~/network_data.yaml anymore) have changed with https://review.openstack.org/#/c/506806/ landing in the puddle so the initial error could be related.


Note You need to log in before you can comment on or make changes to this bug.