Bug 1467398
| Summary: | During RHOSP 9 to RHOSP 10 upgrade, networking-bigswitch is upgraded to Newton release | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | bigswitch <rhosp-bugs-internal> |
| Component: | python-networking-bigswitch | Assignee: | RHOS Maint <rhos-maint> |
| Status: | CLOSED ERRATA | QA Contact: | Ofer Blaut <oblaut> |
| Severity: | urgent | Docs Contact: | |
| Priority: | high | ||
| Version: | 10.0 (Newton) | CC: | amuller, chrisw, jjoyce, jlibosva, jschluet, lhh, mburns, nyechiel, rhosp-bugs-internal, srevivo, tfreger |
| Target Milestone: | z6 | Keywords: | OtherQA, TestOnly, Triaged, ZStream |
| Target Release: | 10.0 (Newton) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | python-networking-bigswitch-9.42.7-1.el7ost | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-11-15 13:53:31 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
sosreport from overcloud controller is uploaded to box https://bigswitch.box.com/s/7hx4b4xndwpxe4wizctarnfnd3k8ta4x Upgrade from the mitaka(osp9) to newton(osp10) version sounds like it's doing the right thing. During the upgrade, all packages get updated to their OSP 10 versions. This sounds a lot like an issue between neutron and the bigswitch plugin doesn't it? Shouldn't this be handled by BigSwitch? Hi Mike, We debugged this further and found the issue but not in a way that we can proceed with upgrade. The Big Switch Networks plugin needs to read properties from under 'keystone_authtoken' in neutron.conf. However, there has been a change in the naming of properties there from Mitaka to Newton. Namely, RHOSP 9 mitaka > RHOSP 10 newton admin_user > username admin_password > password auth_uri > auth_url admin_tenant_name > project_name NA > project_domain_name NA > user_domain_name Now the upgrade process upgrades neutron and networking-bigswitch, but the properties file is still the same old one from Mitaka. So plugin fails to find the properties and neutron-server startup fails. We patched the local env to do it right and neutron-server starts running. But it is not possible to resume upgrade from that state. Can we add some extra_properties in one of the RHOSP related yaml files to specify those properties and make sure they're there when upgrade happens? Either that, or we wait until we patch the Newton version of our plugin and the git tag update reaches the RDO repo which hosts and provides the packages during upgrade. Networking DFG triaged this issue and we've determined the issue can only be fixed in networking-bigswitch code. You'd need to either update networking-bigswitch to use the new conf names in Newton or find a way to find them programatically. Hi Assaf, Apologies for the missing update on the BZ. We have had some communication offline and on mail - the fix for this has been pushed to networking-bigswitch project, such that it recognizes both old and new keys in the conf file. The version with the patch was available before Aug 1 - well in time for the next update of packages for RHOSP 10. I believe the update was scheduled for 15th Aug, but got delayed due to some reason. As soon as the repos are updated with the latest newton package of networking-bigswitch, this would resolve. Do let us know if you have further questions. And also when the RPM repo update happens for RHOSP 10. Thanks! - Aditya You can find the backported fix at python-networking-bigswitch-9.42.7-1.el7ost According to our records, this should be resolved by python-networking-bigswitch-9.42.7-1.el7ost. This build is available now. Thanks Jakub, Lon! Let me give it a try in our setup and update this BZ after verification. - Aditya Any updates on the verification status? Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:3234 The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |
Description of problem: Hi I am trying to upgrade rhosp 9 to rhosp 10. At chapter 3.4.6 (upgrading controller nodes), networking-bigswitch plugin was upgraded from mitaka release to newton release. This causes neutron server to fail to start with the below error [stack@vm-240 ~]$ heat deployment-show 250c8ea5-5502-4938-a852-24a2cbd99c45 WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead { "status": "FAILED", "server_id": "262df95c-b4b3-483d-beb7-4eb205c7bb31", "config_id": "1b23eb12-608f-4d96-8d4c-7b37504fd23a", "output_values": { "deploy_stdout": "Sun Jul 2 21:34:52 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl start httpd\nSun Jul 2 21:34:53 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl enable httpd\nSun Jul 2 21:34:53 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to check_resource_systemd for httpd to be started\nhttpd is started\nSun Jul 2 21:34:54 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl start memcached\nSun Jul 2 21:34:54 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl enable memcached\nSun Jul 2 21:34:54 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to check_resource_systemd for memcached to be started\nmemcached is started\nSun Jul 2 21:34:55 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl start mongod\nSun Jul 2 21:34:55 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl enable mongod\nSun Jul 2 21:34:55 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to check_resource_systemd for mongod to be started\nmongod is started\nSun Jul 2 21:34:55 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl start neutron-dhcp-agent\nSun Jul 2 21:34:56 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl enable neutron-dhcp-agent\nSun Jul 2 21:34:56 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to check_resource_systemd for neutron-dhcp-agent to be started\nneutron-dhcp-agent is started\nSun Jul 2 21:34:57 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl start neutron-l3-agent\nSun Jul 2 21:34:57 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl enable neutron-l3-agent\nSun Jul 2 21:34:57 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to check_resource_systemd for neutron-l3-agent to be started\nneutron-l3-agent is started\nSun Jul 2 21:34:58 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl start neutron-metadata-agent\nSun Jul 2 21:34:58 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl enable neutron-metadata-agent\nSun Jul 2 21:34:58 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to check_resource_systemd for neutron-metadata-agent to be started\nneutron-metadata-agent is started\nSun Jul 2 21:34:59 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl start neutron-netns-cleanup\nSun Jul 2 21:35:00 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl enable neutron-netns-cleanup\nSun Jul 2 21:35:00 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to check_resource_systemd for neutron-netns-cleanup to be started\nneutron-netns-cleanup is started\nSun Jul 2 21:35:00 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl start neutron-openvswitch-agent\nSun Jul 2 21:35:01 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl enable neutron-openvswitch-agent\nSun Jul 2 21:35:01 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to check_resource_systemd for neutron-openvswitch-agent to be started\nneutron-openvswitch-agent is started\nSun Jul 2 21:35:02 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl start neutron-ovs-cleanup\nSun Jul 2 21:35:02 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl enable neutron-ovs-cleanup\nSun Jul 2 21:35:03 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to check_resource_systemd for neutron-ovs-cleanup to be started\nneutron-ovs-cleanup is started\nSun Jul 2 21:35:03 PDT 2017 1b23eb12-608f-4d96-8d4c-7b37504fd23a tripleo-upgrade rhosp9-controller-0 Going to systemctl start neutron-server\n", "deploy_stderr": "Job for neutron-server.service failed because the control process exited with error code. See \"systemctl status neutron-server.service\" and \"journalctl -xe\" for details.\n", "deploy_status_code": 1 }, "creation_time": "2017-07-03T04:34:17Z", "updated_time": "2017-07-03T04:35:02Z", "input_values": { "update_identifier": "1498684273", "deploy_identifier": "1499055564" }, "action": "CREATE", "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 1", "id": "250c8ea5-5502-4938-a852-24a2cbd99c45" } Before the deploy [root@rhosp9-controller-0 heat-admin]# pip show networking-bigswitch --- Metadata-Version: 1.1 Name: networking-bigswitch Version: 8.40.8 Summary: Big Switch Networks Plugins for OpenStack Networking Home-page: http://www.bigswitch.com/ Author: Big Switch Networks Author-email: support License: UNKNOWN Location: /usr/lib/python2.7/site-packages Requires: Classifiers: Environment :: OpenStack Intended Audience :: Information Technology Intended Audience :: System Administrators License :: OSI Approved :: Apache Software License Operating System :: POSIX :: Linux Programming Language :: Python Programming Language :: Python :: 2 Programming Language :: Python :: 2.7 Entry-points: [neutron.service_plugins] bsn_service_plugin = networking_bigswitch.plugins.bigswitch.bsn_service_plugin:BSNServicePlugin bsn_l3 = networking_bigswitch.plugins.bigswitch.l3_router_plugin:L3RestProxy [neutron.db.alembic_migrations] bsn_extensions = networking_bigswitch.plugins.bigswitch.db.migration:alembic_migrations [neutronclient.extension] bsn_extensions = networking_bigswitch.plugins.bigswitch.neutronclient.v2_0._bsn_plugin_client [neutron.ml2.mechanism_drivers] bsn_ml2 = networking_bigswitch.plugins.ml2.drivers.mech_bigswitch.driver:BigSwitchMechanismDriver [console_scripts] neutron-bsn-agent = networking_bigswitch.plugins.bigswitch.agent.restproxy_agent:main bsnlldp = networking_bigswitch.bsnlldp.bsnlldp:main You are using pip version 8.1.2, however version 9.0.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command. [root@rhosp9-controller-0 heat-admin]# After the deploy [root@rhosp9-controller-0 heat-admin]# pip show networking-bigswitch --- Metadata-Version: 1.1 Name: networking-bigswitch Version: 9.42.1 Summary: Big Switch Networks Plugins for OpenStack Networking Home-page: http://www.bigswitch.com/ Author: Big Switch Networks Author-email: support License: UNKNOWN Location: /usr/lib/python2.7/site-packages Requires: Classifiers: Environment :: OpenStack Intended Audience :: Information Technology Intended Audience :: System Administrators License :: OSI Approved :: Apache Software License Operating System :: POSIX :: Linux Programming Language :: Python Programming Language :: Python :: 2 Programming Language :: Python :: 2.7 Entry-points: [console_scripts] bsnlldp = networking_bigswitch.bsnlldp.bsnlldp:main neutron-bsn-agent = networking_bigswitch.plugins.bigswitch.agent.restproxy_agent:main [neutron.db.alembic_migrations] bsn_extensions = networking_bigswitch.plugins.bigswitch.db.migration:alembic_migrations [neutron.ml2.mechanism_drivers] bsn_ml2 = networking_bigswitch.plugins.ml2.drivers.mech_bigswitch.driver:BigSwitchMechanismDriver [neutron.service_plugins] bsn_l3 = networking_bigswitch.plugins.bigswitch.l3_router_plugin:L3RestProxy bsn_service_plugin = networking_bigswitch.plugins.bigswitch.bsn_service_plugin:BSNServicePlugin [neutronclient.extension] bsn_extensions = networking_bigswitch.plugins.bigswitch.neutronclient.v2_0._bsn_plugin_client You are using pip version 8.1.2, however version 9.0.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command. [root@rhosp9-controller-0 heat-admin]# Version-Release number of selected component (if applicable): RHOSP 10 How reproducible: Always This is the deploy command time openstack overcloud deploy --stack rhosp9 -e /home/stack/templates/bigswitch-config.yaml -e /home/stack/templates/network-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /home/stack/templates/timezone.yaml -e /home/stack/templates/rhel-registration/environment-rhel-registration.yaml -e /home/stack/templates/rhel-registration/rhel-registration-resource-registry.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker.yaml --neutron-network-type vlan --neutron-network-vlan-ranges rhosp:1001:2000 --neutron-bridge-mappings "rhosp:br-ex" --neutron-disable-tunneling --compute-scale 2 --control-scale 1 --templates --control-flavor control --compute-flavor compute --ntp-server 0.rhel.pool.ntp.org --timeout $timeout --debug 2>&1 | tee haha-$date.log