Bug 1666598 - I need a fix for this one
Summary: I need a fix for this one
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 14.0 (Rocky)
Hardware: x86_64
OS: Linux
unspecified
low
Target Milestone: ---
: ---
Assignee: Brian Haley
QA Contact: Roee Agiman
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-01-16 07:19 UTC by Amer Hwitat
Modified: 2019-01-28 14:58 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
Clone Of:
Environment:
ubuntu
Last Closed: 2019-01-28 14:58:43 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
neutron bug (101.99 KB, image/jpeg)
2019-01-16 07:19 UTC, Amer Hwitat
no flags Details

Description Amer Hwitat 2019-01-16 07:19:28 UTC
Created attachment 1520882 [details]
neutron bug

Description of problem:
it seems that Red Hat Openstack 13 has problems implementing Virtual Bridges .. it affects Neutron service (Networking) on Openstack , Network interfaces on the Routers should not be DOWN state ... see attached


Version-Release number of selected component (if applicable):

Openstack 14

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Amer Hwitat 2019-01-16 15:30:52 UTC
I have discussed this with Brian on :

https://bugs.launchpad.net/neutron/+bug/1811941

he told me that Bugzilla knows better in these issues I issued the post with complete commands I gave there.

please read the below is my correspondence with Brian:
sorry I'm using Red Hat 7.0

Hide
Brian Haley (brian-haley) wrote 58 minutes ago:	#3
First, if this is a bug specific to OSP, you should file the bug in Red Hat bugzilla.

Second, what commands did you run to create and configure the router? The jpg doesn't show anything except a router connected to the external network (maybe there is a private subnet hiding behind the pop-up?).

Changed in neutron:
status:	New → Incomplete
Amer Hwitat (amer.hwitat) wrote 35 minutes ago:	#4
Download full text (3.4 KiB)
I used this to install on RHEL 7
########################################################################
systemctl disable NetworkManager
systemctl stop NetworkManager
systemctl disable firewalld
systemctl stop firewalld
setenforce 0

systemctl restart network
systemctl status network
########################################################################
subscription-manager list --available

subscription-manager attach --pool=

subscription-manager repos --enable=rhel-7-server-optional-rpms \
  --enable=rhel-7-server-extras-rpms --enable=rhel-7-server-rh-common-rpms
subscription-manager repos --enable=rhel-7-server-openstack-14-rpms
subscription-manager repos --enable=rhel-7-server-openstack-14-devtools-rpms
subscription-manager repos --enable=rhel-7-server-openstack-14-tools-rpms

yum repolist enabled #enable all

subscriptiion-manager repos --enable=

yum install -y yum-plugin-priorities yum-utils

yum install openstack-selinux

rpm -q --whatprovides rubygem-json ###### rubygem-json-1.7.7-20.el7.x86_64

yum install -y openstack-packstack
############################################################################
I used this command to create external network:
neutron net-create External1 --provider:network_type flat --provider:physical_network br-ex --router:external=true --shared
neutron net-create External2 --provider-physical-network provider --provider:physical_network eno16777736 --router:external=true --shared
openstack subnet create --network provider \
  --allocation-pool start=192.168.43.1,end=192.168.43.240 \
  --dns-nameserver 192.168.43.1 --gateway 192.168.43.1 \
  --subnet-range 192.168.43.0/24 provider
I tried the three external networks separately
#############################################################################
I created the routers and subnets on Horizon

my external network is 192.168.43.0/24 my IP is 192.168.43.77 I used for installing openstack packstack with answer file and https enabled

I also used this solution and configured br-ex on my network for openstack also configuered related neutron .ini and plugins .ini files

I also used this solution:
mysql
  > create database neutron;
  > grant all privileges on neutron.* to 'neutron'@'localhost' identified by 'server';
  > grant all privileges on neutron.* to 'neutron'@'%' identified by 'server';
  > quit

export | grep OS_declare -x OS_AUTH_URL="https://192.168.43.77:5000/v3"

source admin-openrc.sh
openstack user create --domain default --password-prompt neutron
openstack role add --project service --user neutron admin
openstack service create --name neutron --description "OpenStack Networking" network
openstack endpoint create --region RegionOne network public http://controller:9696
openstack endpoint create --region RegionOne network internal http://controller:9696
openstack endpoint create --region RegionOne network admin http://controller:9696

systemctl enable neutron-server.service neutron-openvswitch-agent.service neutron-dhcp-agent.service neutron-metadata-agent.service neutron-ovs-cleanup.service
systemctl start neutron-server.service neutron-openvswitch-agent.service neutron-dhcp-agent.service neutron-metadata-agent.service neutron-ovs-cle...

Read more...

Hide
Amer Hwitat (amer.hwitat) wrote 31 minutes ago:	#5
I also followed the following links for answer:

https://www.linuxtechi.com/install-use-openvswitch-kvm-centos-7-rhel-7/

https://ask.openstack.org/en/question/109367/how-to-debug-the-routers-interface-all-the-interfaces-status-are-down/?answer=118926#post-id-118926

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/networking_guide/sec-connect-instance

https://ask.openstack.org/en/question/25234/one-router-port-is-always-down/

https://openstack-xenserver.readthedocs.io/en/latest/10-install-networking-neutron-on-controller.html

https://bugzilla.redhat.com/show_bug.cgi?id=1054857

Created a Bridge also using:

ovs-vsctl add-br br-ex
ip addr add 192.168.43.77/24 dev br-ex
ip addr flush dev eno16777736
ip addr add 192.168.43.77/24 dev br-ex
ovs-vsctl add-port br-ex eno16777736
ip link set dev br-ex up

virsh net-define /tmp/ovs-network.xml \
Network ovs-network defined from /tmp/ovs-network.xml
########################################################################

Hide
Amer Hwitat (amer.hwitat) wrote 20 minutes ago:	#6
Ohh yes there is an internal network configured as 172.17.0.0/16 and it's internal network also the interface created using Horizon and was Down, for the External network it's created by CLI, as I didn't find this option in Horizon for Red Hat Openstack 13 ... :) excuse me , I didn't get so much sleep today and yesterday trying to workaround this problem, I also sent the Red Hat about this problem on Buzilla after you mentioned it, thanks to you ...

Hide
Amer Hwitat (amer.hwitat) wrote 19 minutes ago:	#7
Both networks are Active on Horizon (External, and Internal), but the interfaces is DOWN ...

Hide
Brian Haley (brian-haley) wrote 4 minutes ago:	#8
Your instructions above do not show you starting the l3-agent, which will wire and bring the router interfaces up.

> systemctl enable neutron-server.service neutron-openvswitch-agent.service neutron-dhcp-agent.service neutron-metadata-agent.service neutron-ovs-cleanup.service
> systemctl start neutron-server.service neutron-openvswitch-agent.service neutron-dhcp-agent.service neutron-metadata-agent.service neutron-ovs-cleanup.service

$ systemctl enable neutron-l3-agent.service
$ systemctl start neutron-l3-agent.service

This is listed in the next block in the install guide, https://docs.openstack.org/neutron/latest/install/controller-install-rdo.html

As a data point, issues like this are best discussed on openstack.org since they aren't always bugs.

Amer Hwitat (amer.hwitat) wrote 3 minutes ago:	#9
I'm thinking where to post the Snapshots of this problem for you

Hide
See full activity log
Amer Hwitat (amer.hwitat) wrote a moment ago:	#10
well I did enable it as you said earlier but it didn't help before ... I forgot to mention it in the answer sheet I posted first sorry, but I want to ask,if there is an option to create External netwroks on Horizon as it seems there isn't any, I mean in RH Openstack, unlike RDO Openstack, anyway, my VM crashed and I deleted it from my HD, it seems that I'm using it on my laptop as allinone you know with 5 GB of RAM which slows me down, I'm doing this for testing purposes and started another fresh installation now, my new network, does it really needs a Bridge , I mean br-ex to work properly, or I might use eno16777736 interface for external network directly.

Thanks Brian

Comment 2 Amer Hwitat 2019-01-16 15:31:32 UTC
if there is a place to post the snapshots it would be great

Comment 3 Amer Hwitat 2019-01-16 15:49:41 UTC
the new installation gave :
ERROR : Error appeared during Puppet run: 192.168.64.128_controller.pp
Error: Execution of '/usr/bin/yum -d 0 -e 0 -y install openstack-neutron' returned 1: No Presto metadata available for rhel-7-server-rpms
You will find full trace in log /var/tmp/packstack/20190116-095337-ptPHfw/manifests/192.168.64.128_controller.pp.log

I think there is something here, I added:

subscription-manager repos --enable=rhel-7-server-openstack-13-optools-rpms
subscription-manager repos --enable=rhel-7-server-openstack-13-rpms
subscription-manager repos --enable=rhel-7-server-openstack-13-devtools-rpms
subscription-manager repos --enable=rhel-7-server-openstack-13-tools-rpms

to the repos list

I'm now tailing (tail -f) the log file for the new packstack --answer-file= my answer

I will give you feed back on this issue again later ...

but the question for Brian stands for you please, do I need to configure the br-ex bridge to let neutron work and why there isn't any External networks option in Horizon for RH Openstack, also I'm on 60 days trial for Openstack 14, how can I install it --allinone

Best of all, thanks

Comment 4 Amer Hwitat 2019-01-16 15:53:48 UTC
it ran smoothly and it's in the check-update phase

Comment 5 Amer Hwitat 2019-01-16 15:59:34 UTC
ps -ef|grep -i neutron
root 113380 108515 9 10:54 ? 00:00:02 /usr/bin/python2 /usr/bin/neutron-db-manage upgrade heads

it's upgrading I think it will run and I will do the External and Internal check with a router on Horizon again

Thanks

Comment 6 Brian Haley 2019-01-16 18:46:34 UTC
Amer - I still don't think this is an issue with OSP 14 as we do not see this problem in our deployments.

We can keep the discussion in the upstream bug until we can figure out what is mis-configured in your deployment.

Comment 7 Amer Hwitat 2019-01-16 19:43:55 UTC
definitely I agree that's why I'm checking and learning, thanks for the support Brian, I will do it tonight,but again the laptop crashed because of Memory heavy use :)

Comment 8 Amer Hwitat 2019-01-16 22:25:29 UTC
the problem is with the installation timeout=0 and --debug mode I'm doing that now

Comment 9 Amer Hwitat 2019-01-16 22:30:25 UTC
I issued a packstack --answer-file=/root/answer.txt --timeout=300 --debug to solve it it's reinstalling now

Comment 10 Amer Hwitat 2019-01-17 00:10:16 UTC
https://access.redhat.com/errata/RHBA-2018:0731

I installed updates

yum update ### it's 1.3 it will upgrade to maipo 7.6

maybe this will make it better :)

Comment 11 Amer Hwitat 2019-01-17 02:18:57 UTC
https://access.redhat.com/errata/RHEA-2019:0045

Topic

Red Hat OpenStack Platform 14.0 (Rocky) for RHEL 7.

Description

Red Hat OpenStack Platform provides the facilities for building, deploying
and monitoring a private or public infrastructure-as-a-service (IaaS) cloud
running on commonly available physical hardware.

For more information about the bug fixes and enhancements included with this update, see the "Technical Notes" section of the Release Notes linked in the References section.

Solution

Before applying this update, ensure all previously released errata relevant
to your system have been applied.

Red Hat OpenStack Platform 14 runs on Red Hat Enterprise Linux 7.5.

The Red Hat OpenStack Platform 14 Release Notes contain the following:

    An explanation of the way in which the provided components interact to
    form a working cloud computing environment.
    Technology Previews, Recommended Practices, and Known Issues.
    The channels required for Red Hat OpenStack Platform 14, including which
    channels need to be enabled and disabled.

The Release Notes are available at:
https://access.redhat.com/documentation/en/red-hat-openstack-platform/

This update is available through 'yum update' on systems registered through
Red Hat Subscription Manager. For more information about Red Hat
Subscription Manager, see:

https://access.redhat.com/documentation/en-US/Red_Hat_Subscription_Management/1/html/RHSM/index.html

Affected Products

    Red Hat Enterprise Linux Server 7 x86_64
    Red Hat Enterprise Linux Workstation 7 x86_64
    Red Hat Enterprise Linux Desktop 7 x86_64
    Red Hat OpenStack 14 x86_64
    Red Hat OpenStack for IBM Power 14 ppc64le
    Red Hat OpenStack Director Deployment Tools 14 x86_64
    Red Hat OpenStack Director Deployment Tools for IBM Power LE 14 ppc64le
references:


    https://access.redhat.com/documentation/en-us/red_hat_openstack_platform
###############################################################################
so I have disabled and enabled:
subscription-manager repos --enable=rhel-7-server-optional-rpms \
  --enable=rhel-7-server-extras-rpms --enable=rhel-7-server-rh-common-rpms
subscription-manager repos --disable=rhel-7-server-openstack-14-rpms
subscription-manager repos --disable=rhel-7-server-openstack-14-devtools-rpms
subscription-manager repos --disable=rhel-7-server-openstack-14-tools-rpms

and then

yum update

after that 

yum install -y openstack-packstack

then

packstack --answer-file=/root/answer.txt

it takes lots of time with 5B VM RAM ....

windows 10 EN sucks but I need it to teach in the Lab you know...

but you didn't say , why there is no External Newtwork creation in Horizon and I have to issue a command line to create the Netwrok, and it's a deprecated command in RH Openstack 14 ???

also why RH people do not make auto update for the system you have to do it manually CLI ...

Thanks for your support ...

Comment 12 Amer Hwitat 2019-01-17 02:48:05 UTC
I renabled the openstack 14 repos

it still says that it's installing openstack 13 components 

that's interesting

Comment 13 Amer Hwitat 2019-01-17 04:23:23 UTC
Hi Brian ...

in case you are not aware of what has happened to my installation of RHEL 7.0

I'm now in the process of updating it ... it takes time ... Red Hat openstack 14 rocky works on 7.5 upgrades of 7.0 RHEL only ... I don't know how I managed to install it on RHEL 7.0 in the fist installation ... but this seems to be the problem ... the urllib3 and chadet libraries of python gives compatibility issue ... although I'm not sure how the installation was the first time success ....

I payed attention to that in the second installation and third ... this is my fourth installation ... also takes time ...

I'm anxious to complete the upgrade (yum update or upgrade the same really on RHEL 7.0)

so that I can install and comfigure openstack 14 rocky ... it's a good IaaS open source solution for me ...

I'm doing this as a favor  to my friend in a Big Big Telecom company in Saudi ... they are implementing RH Openstack as IaaS platform to mount over it some services ... and I'm just providing help to my friend, though it's interesting and might benefit from also ... I like open source things it's really good ... Linux is hot tooooooooo  much .. as an Old MCSE 2000 certified Engineer I say it ... I like it very much ... what I don't like is speaking to the open air and landscape instead of people in Bugzilla ... I didn't get a reply yet ... that's why I'm writing to you here ...

Once finished from installation I will write back on the results on neutron ... though I might give credit to RH honestly asking me how is my experience in Openstack 14, I don't know how to install it --allinone ;) they called last week and I'm on 60 days trial ... I want to finish before that evaluation ...

I'm not in the process of installing undercloud director and overcloud ... I want to evaluate the services on one VM and specially Horizon Nova Neutron Cinder swift Glance and heat (Sahara and Heat really)

thanks

Comment 14 Amer Hwitat 2019-01-17 16:32:50 UTC
I have figured it out that the issue was a trouble in installation... so what I did I opened three putty sessions to monitor the installation after I figured out from my subscription that I used 2 licenses out of 18 for Openstack 14 and RHEL 7.6 for each ... it was a long day ... I'm still troubleshooting ... I want everything to install right .. so I monitored top in the first session and ps -ef|grep openstack in the second and the log of openstack-setup in debug mode of packstack --debug I ran on the VM itself ... so I'm monitoring 4 sessions really 

it is still installing the packages ... I payed attention that it finished neutron where I want to troubleshoot, but still don't know if it finished configuration of openstack neutron DB entries ...

I will try to finish today and create the Networks and router on CLI ...

Thanks

Comment 15 Amer Hwitat 2019-01-17 19:35:19 UTC
export OS_USERNAME=admin
export OS_PASSWORD=server
export OS_TENANT_NAME=admin
export OS_AUTH_URL=http://localhost:5000/v3

export OS_USERNAME=admin
export OS_PASSWORD=server
export OS_PROJECT_NAME=project
export OS_USER_DOMAIN_ID=default
export OS_PROJECT_DOMAIN_ID=default
export OS_AUTH_URL=http://auth.example.com:5000/v3

neutron router-gateway-set ROUTER NETWORK
neutron router-interface-add ROUTER SUBNET

neutron router-gateway-set router1 External2

neutron net-create External2 --provider:network_type flat --provider:physical_network eno16777736 --router:external=true --shared
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Created a new network:
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| availability_zone_hints   |                                      |
| availability_zones        |                                      |
| created_at                | 2019-01-17T17:05:57Z                 |
| description               |                                      |
| id                        | 0f9adf89-6ee2-4e68-8559-fc474dedb30d |
| ipv4_address_scope        |                                      |
| ipv6_address_scope        |                                      |
| is_default                | False                                |
| mtu                       | 1500                                 |
| name                      | External2                            |
| port_security_enabled     | True                                 |
| project_id                | a76eea958a6e435a93e3ffc7a36c7970     |
| provider:network_type     | flat                                 |
| provider:physical_network | eno16777736                          |
| provider:segmentation_id  |                                      |
| qos_policy_id             |                                      |
| revision_number           | 1                                    |
| router:external           | True                                 |
| shared                    | True                                 |
| status                    | ACTIVE                               |
| subnets                   |                                      |
| tags                      |                                      |
| tenant_id                 | a76eea958a6e435a93e3ffc7a36c7970     |
| updated_at                | 2019-01-17T17:05:57Z                 |
+---------------------------+--------------------------------------+
[root@localhost ~(keystone_admin)]# neutron router-create router1
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Created a new router:
+-------------------------+--------------------------------------+
| Field                   | Value                                |
+-------------------------+--------------------------------------+
| admin_state_up          | True                                 |
| availability_zone_hints |                                      |
| availability_zones      |                                      |
| created_at              | 2019-01-17T17:09:36Z                 |
| description             |                                      |
| distributed             | False                                |
| external_gateway_info   |                                      |
| flavor_id               |                                      |
| ha                      | False                                |
| id                      | 59dabea6-0f82-48e4-a093-9a4a904df496 |
| name                    | router1                              |
| project_id              | a76eea958a6e435a93e3ffc7a36c7970     |
| revision_number         | 0                                    |
| routes                  |                                      |
| status                  | ACTIVE                               |
| tags                    |                                      |
| tenant_id               | a76eea958a6e435a93e3ffc7a36c7970     |
| updated_at              | 2019-01-17T17:09:36Z                 |
+-------------------------+--------------------------------------+
[root@localhost ~(keystone_admin)]# neutron router-gateway-set router1 External2
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Set gateway for router router1
[root@localhost ~(keystone_admin)]# ^C
[root@localhost ~(keystone_admin)]# neutron router-interface-add router1 Internal
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Added interface 1271621e-4cf0-425e-a6a8-ad68c3b43c7a to router router1.

neutron port-create Internal --fixed-ip ip_address=172.17.20.5
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Created a new port:
+-----------------------+------------------------------------------------------------------------------------+
| Field                 | Value                                                                              |
+-----------------------+------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                               |
| allowed_address_pairs |                                                                                    |
| binding:host_id       |                                                                                    |
| binding:profile       | {}                                                                                 |
| binding:vif_details   | {}                                                                                 |
| binding:vif_type      | unbound                                                                            |
| binding:vnic_type     | normal                                                                             |
| created_at            | 2019-01-17T17:19:52Z                                                               |
| description           |                                                                                    |
| device_id             |                                                                                    |
| device_owner          |                                                                                    |
| extra_dhcp_opts       |                                                                                    |
| fixed_ips             | {"subnet_id": "4f61307a-6f89-4edc-8c9b-f4afc9c9bde6", "ip_address": "172.17.20.5"} |
| id                    | b2639fd0-a2ff-4616-9f69-1a86240a65ce                                               |
| mac_address           | fa:16:3e:ff:0c:d1                                                                  |
| name                  |                                                                                    |
| network_id            | 5f3ec4e3-a5c4-4b0b-8e9e-add01ffa2960                                               |
| port_security_enabled | True                                                                               |
| project_id            | a76eea958a6e435a93e3ffc7a36c7970                                                   |
| qos_policy_id         |                                                                                    |
| revision_number       | 2                                                                                  |
| security_groups       | 0edb18c8-e7ce-4f6f-9a81-2a24b9a8769f                                               |
| status                | DOWN                                                                               |
| tags                  |                                                                                    |
| tenant_id             | a76eea958a6e435a93e3ffc7a36c7970                                                   |
| updated_at            | 2019-01-17T17:19:52Z                                                               |
+-----------------------+------------------------------------------------------------------------------------+


neutron port-create Internal
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Created a new port:
+-----------------------+--------------------------------------------------------------------------------------+
| Field                 | Value                                                                                |
+-----------------------+--------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                 |
| allowed_address_pairs |                                                                                      |
| binding:host_id       |                                                                                      |
| binding:profile       | {}                                                                                   |
| binding:vif_details   | {}                                                                                   |
| binding:vif_type      | unbound                                                                              |
| binding:vnic_type     | normal                                                                               |
| created_at            | 2019-01-17T17:21:59Z                                                                 |
| description           |                                                                                      |
| device_id             |                                                                                      |
| device_owner          |                                                                                      |
| extra_dhcp_opts       |                                                                                      |
| fixed_ips             | {"subnet_id": "4f61307a-6f89-4edc-8c9b-f4afc9c9bde6", "ip_address": "172.17.20.152"} |
| id                    | 53c7f03e-27b6-419d-a098-61823dad013b                                                 |
| mac_address           | fa:16:3e:94:b5:b2                                                                    |
| name                  |                                                                                      |
| network_id            | 5f3ec4e3-a5c4-4b0b-8e9e-add01ffa2960                                                 |
| port_security_enabled | True                                                                                 |
| project_id            | a76eea958a6e435a93e3ffc7a36c7970                                                     |
| qos_policy_id         |                                                                                      |
| revision_number       | 2                                                                                    |
| security_groups       | 0edb18c8-e7ce-4f6f-9a81-2a24b9a8769f                                                 |
| status                | DOWN                                                                                 |
| tags                  |                                                                                      |
| tenant_id             | a76eea958a6e435a93e3ffc7a36c7970                                                     |
| updated_at            | 2019-01-17T17:22:00Z                                                                 |
+-----------------------+--------------------------------------------------------------------------------------+


neutron port-list --fixed-ips ip_address=192.168.43.1   ip_address=172.17.20.5 ip_address=172.17.20.152
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+--------------------------------------+------+----------------------------------+-------------------+--------------------------------------------------------------------------------------+
| id                                   | name | tenant_id                        | mac_address       | fixed_ips                                                                            |
+--------------------------------------+------+----------------------------------+-------------------+--------------------------------------------------------------------------------------+
| 53c7f03e-27b6-419d-a098-61823dad013b |      | a76eea958a6e435a93e3ffc7a36c7970 | fa:16:3e:94:b5:b2 | {"subnet_id": "4f61307a-6f89-4edc-8c9b-f4afc9c9bde6", "ip_address": "172.17.20.152"} |
| b2639fd0-a2ff-4616-9f69-1a86240a65ce |      | a76eea958a6e435a93e3ffc7a36c7970 | fa:16:3e:ff:0c:d1 | {"subnet_id": "4f61307a-6f89-4edc-8c9b-f4afc9c9bde6", "ip_address": "172.17.20.5"}   |
+--------------------------------------+------+----------------------------------+-------------------+--------------------------------------------------------------------------------------+


neutron router-update router1 --routes type=dict list=true destination=192.168.43.0/24,nexthop=172.17.20.150
neutron router-update router1 --routes type=dict list=true destination=172.17.0.0,nexthop=192.168.43.1

Comment 16 Amer Hwitat 2019-01-17 19:48:22 UTC
yum install -y *openvswitch*

I'm waiting installing rhosp-openvswitch*

it's working now ...

I don't know if it was the packstack fault or mine now ... :0

yum install -y *openvswitch*
Loaded plugins: langpacks, priorities, product-id, search-disabled-repos,
              : subscription-manager
Package python-openvswitch2.10-2.10.0-28.el7fdp.1.x86_64 already installed and latest version
Package openvswitch-selinux-extra-policy-1.0-9.el7fdp.noarch already installed and latest version
Package openvswitch-ovn-host is obsoleted by rhosp-openvswitch-ovn-host, trying to install rhosp-openvswitch-ovn-host-2.10-0.1.el7ost.noarch instead
Package openvswitch2.10-2.10.0-28.el7fdp.1.x86_64 already installed and latest version
Package openvswitch-ovn-common is obsoleted by rhosp-openvswitch-ovn-common, trying to install rhosp-openvswitch-ovn-common-2.10-0.1.el7ost.noarch instead
Package openvswitch-test is obsoleted by rhosp-openvswitch-test, trying to install rhosp-openvswitch-test-2.10-0.1.el7ost.noarch instead
Package python-rhosp-openvswitch-2.10-0.1.el7ost.noarch already installed and latest version
Package openvswitch is obsoleted by rhosp-openvswitch, trying to install rhosp-openvswitch-2.10-0.1.el7ost.noarch instead
Package openvswitch-ovn-vtep is obsoleted by rhosp-openvswitch-ovn-vtep, trying to install rhosp-openvswitch-ovn-vtep-2.10-0.1.el7ost.noarch instead
Package openvswitch-ovn-central is obsoleted by rhosp-openvswitch-ovn-central, trying to install rhosp-openvswitch-ovn-central-2.10-0.1.el7ost.noarch instead
Package python-openvswitch-2.9.0-83.el7fdp.1.x86_64 is obsoleted by python-rhosp-openvswitch-2.10-0.1.el7ost.noarch which is already installed
Package openvswitch-controller-2.0.0-7.el7.x86_64 is obsoleted by openvswitch2.10-2.10.0-28.el7fdp.1.x86_64 which is already installed
Resolving Dependencies
--> Running transaction check
---> Package openstack-neutron-openvswitch.noarch 1:13.0.2-0.20180929022427.c7f970c.el7ost will be installed
---> Package openvswitch2.10-devel.x86_64 0:2.10.0-28.el7fdp.1 will be installed
---> Package openvswitch2.10-ovn-central.x86_64 0:2.10.0-28.el7fdp.1 will be installed
---> Package openvswitch2.10-ovn-common.x86_64 0:2.10.0-28.el7fdp.1 will be installed
---> Package openvswitch2.10-ovn-host.x86_64 0:2.10.0-28.el7fdp.1 will be installed
---> Package openvswitch2.10-ovn-vtep.x86_64 0:2.10.0-28.el7fdp.1 will be installed
---> Package openvswitch2.10-test.noarch 0:2.10.0-28.el7fdp.1 will be installed
--> Processing Dependency: python-twisted-core for package: openvswitch2.10-test-2.10.0-28.el7fdp.1.noarch
--> Processing Dependency: python-twisted-web for package: openvswitch2.10-test-2.10.0-28.el7fdp.1.noarch
---> Package rhosp-openvswitch.noarch 0:2.10-0.1.el7ost will be installed
---> Package rhosp-openvswitch-devel.noarch 0:2.10-0.1.el7ost will be installed
---> Package rhosp-openvswitch-ovn-central.noarch 0:2.10-0.1.el7ost will be installed
---> Package rhosp-openvswitch-ovn-common.noarch 0:2.10-0.1.el7ost will be installed
---> Package rhosp-openvswitch-ovn-host.noarch 0:2.10-0.1.el7ost will be installed
---> Package rhosp-openvswitch-ovn-vtep.noarch 0:2.10-0.1.el7ost will be installed
---> Package rhosp-openvswitch-test.noarch 0:2.10-0.1.el7ost will be installed
--> Running transaction check
---> Package python-twisted.x86_64 0:16.1.1-5.1.el7ost will be installed
--> Processing Dependency: python-zope-interface >= 3.6.0 for package: python-twisted-16.1.1-5.1.el7ost.x86_64
--> Processing Dependency: pyserial for package: python-twisted-16.1.1-5.1.el7ost.x86_64
--> Processing Dependency: python-service-identity for package: python-twisted-16.1.1-5.1.el7ost.x86_64
--> Running transaction check
---> Package pyserial.noarch 0:2.6-6.el7 will be installed
---> Package python-service-identity.noarch 0:14.0.0-4.el7ost will be installed
--> Processing Dependency: python-characteristic for package: python-service-identity-14.0.0-4.el7ost.noarch
---> Package python-zope-interface.x86_64 0:4.0.5-4.el7 will be installed
--> Running transaction check
---> Package python-characteristic.noarch 0:14.3.0-4.el7ost will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package        Arch   Version            Repository                       Size
================================================================================
Installing:
 openstack-neutron-openvswitch
                noarch 1:13.0.2-0.20180929022427.c7f970c.el7ost
                                          rhel-7-server-openstack-14-rpms  16 k
 openvswitch2.10-devel
                x86_64 2.10.0-28.el7fdp.1 rhel-7-server-openstack-14-rpms 157 k
 openvswitch2.10-ovn-central
                x86_64 2.10.0-28.el7fdp.1 rhel-7-server-openstack-14-rpms 1.0 M
 openvswitch2.10-ovn-common
                x86_64 2.10.0-28.el7fdp.1 rhel-7-server-openstack-14-rpms 2.8 M
 openvswitch2.10-ovn-host
                x86_64 2.10.0-28.el7fdp.1 rhel-7-server-openstack-14-rpms 1.0 M
 openvswitch2.10-ovn-vtep
                x86_64 2.10.0-28.el7fdp.1 rhel-7-server-openstack-14-rpms 939 k
 openvswitch2.10-test
                noarch 2.10.0-28.el7fdp.1 rhel-7-server-openstack-14-devtools-rpms
                                                                           68 k
 rhosp-openvswitch
                noarch 2.10-0.1.el7ost    rhel-7-server-openstack-14-rpms 3.0 k
 rhosp-openvswitch-devel
                noarch 2.10-0.1.el7ost    rhel-7-server-openstack-14-rpms 3.0 k
 rhosp-openvswitch-ovn-central
                noarch 2.10-0.1.el7ost    rhel-7-server-openstack-14-rpms 3.0 k
 rhosp-openvswitch-ovn-common
                noarch 2.10-0.1.el7ost    rhel-7-server-openstack-14-rpms 3.0 k
 rhosp-openvswitch-ovn-host
                noarch 2.10-0.1.el7ost    rhel-7-server-openstack-14-rpms 3.0 k
 rhosp-openvswitch-ovn-vtep
                noarch 2.10-0.1.el7ost    rhel-7-server-openstack-14-rpms 3.0 k
 rhosp-openvswitch-test
                noarch 2.10-0.1.el7ost    rhel-7-server-openstack-14-devtools-rpms
                                                                          3.0 k
Installing for dependencies:
 pyserial       noarch 2.6-6.el7          rhel-7-server-rpms              124 k
 python-characteristic
                noarch 14.3.0-4.el7ost    rhel-7-server-openstack-13-rpms  30 k
 python-service-identity
                noarch 14.0.0-4.el7ost    rhel-7-server-openstack-13-rpms  19 k
 python-twisted x86_64 16.1.1-5.1.el7ost  rhel-7-server-openstack-13-rpms 5.5 M
 python-zope-interface
                x86_64 4.0.5-4.el7        rhel-7-server-openstack-13-rpms 138 k

Transaction Summary
================================================================================
Install  14 Packages (+5 Dependent packages)

Total download size: 12 M
Installed size: 47 M
Downloading packages:
(1/19): openstack-neutron-openvswitch-13.0.2-0.20180929022 |  16 kB   00:02     
(2/19): openvswitch2.10-devel-2.10.0-28.el7fdp.1.x86_64.rp | 157 kB   00:02     
(3/19): openvswitch2.10-ovn-central-2.10.0-28.el7fdp.1.x86 | 1.0 MB   00:03     
(4/19): openvswitch2.10-ovn-host-2.10.0-28.el7fdp.1.x86_64 | 1.0 MB   00:03     
(5/19): openvswitch2.10-ovn-common-2.10.0-28.el7fdp.1.x86_ | 2.8 MB   00:06     
(6/19): python-service-identity-14.0.0-4.el7ost.noarch.rpm |  19 kB   00:01     
(7/19): pyserial-2.6-6.el7.noarch.rpm                      | 124 kB   00:01     
(8/19): openvswitch2.10-test-2.10.0-28.el7fdp.1.noarch.rpm |  68 kB   00:01     
(9/19): python-characteristic-14.3.0-4.el7ost.noarch.rpm   |  30 kB   00:02     
(10/19): openvswitch2.10-ovn-vtep-2.10.0-28.el7fdp.1.x86_6 | 939 kB   00:02     
(11/19): rhosp-openvswitch-devel-2.10-0.1.el7ost.noarch.rp | 3.0 kB   00:00     
(12/19): rhosp-openvswitch-2.10-0.1.el7ost.noarch.rpm      | 3.0 kB   00:01     
(13/19): python-zope-interface-4.0.5-4.el7.x86_64.rpm      | 138 kB   00:01     
(14/19): rhosp-openvswitch-ovn-central-2.10-0.1.el7ost.noa | 3.0 kB   00:01     
(15/19): rhosp-openvswitch-ovn-common-2.10-0.1.el7ost.noar | 3.0 kB   00:01     
(16/19): rhosp-openvswitch-ovn-host-2.10-0.1.el7ost.noarch | 3.0 kB   00:01     
(17/19): rhosp-openvswitch-ovn-vtep-2.10-0.1.el7ost.noarch | 3.0 kB   00:01     
(18/19): rhosp-openvswitch-test-2.10-0.1.el7ost.noarch.rpm | 3.0 kB   00:01     
(19/19): python-twisted-16.1.1-5.1.el7ost.x86_64.rpm       | 5.5 MB   00:06     
--------------------------------------------------------------------------------
Total                                              686 kB/s |  12 MB  00:17     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : openvswitch2.10-ovn-common-2.10.0-28.el7fdp.1.x86_64        1/19 
  Installing : openvswitch2.10-ovn-host-2.10.0-28.el7fdp.1.x86_64          2/19 
  Installing : openvswitch2.10-ovn-central-2.10.0-28.el7fdp.1.x86_64       3/19 
  Installing : openvswitch2.10-ovn-vtep-2.10.0-28.el7fdp.1.x86_64          4/19 
  Installing : openvswitch2.10-devel-2.10.0-28.el7fdp.1.x86_64             5/19 
  Installing : python-zope-interface-4.0.5-4.el7.x86_64                    6/19 
  Installing : python-characteristic-14.3.0-4.el7ost.noarch                7/19 
  Installing : python-service-identity-14.0.0-4.el7ost.noarch              8/19 
  Installing : rhosp-openvswitch-2.10-0.1.el7ost.noarch                    9/19 
  Installing : pyserial-2.6-6.el7.noarch                                  10/19 
  Installing : python-twisted-16.1.1-5.1.el7ost.x86_64                    11/19 
  Installing : openvswitch2.10-test-2.10.0-28.el7fdp.1.noarch             12/19 
  Installing : rhosp-openvswitch-test-2.10-0.1.el7ost.noarch              13/19 
  Installing : 1:openstack-neutron-openvswitch-13.0.2-0.20180929022427.   14/19 
  Installing : rhosp-openvswitch-devel-2.10-0.1.el7ost.noarch             15/19 
  Installing : rhosp-openvswitch-ovn-vtep-2.10-0.1.el7ost.noarch          16/19 
  Installing : rhosp-openvswitch-ovn-central-2.10-0.1.el7ost.noarch       17/19 
  Installing : rhosp-openvswitch-ovn-host-2.10-0.1.el7ost.noarch          18/19 
  Installing : rhosp-openvswitch-ovn-common-2.10-0.1.el7ost.noarch        19/19 
  Verifying  : pyserial-2.6-6.el7.noarch                                   1/19 
  Verifying  : rhosp-openvswitch-test-2.10-0.1.el7ost.noarch               2/19 
  Verifying  : openvswitch2.10-ovn-host-2.10.0-28.el7fdp.1.x86_64          3/19 
  Verifying  : rhosp-openvswitch-ovn-common-2.10-0.1.el7ost.noarch         4/19 
  Verifying  : openvswitch2.10-ovn-central-2.10.0-28.el7fdp.1.x86_64       5/19 
  Verifying  : rhosp-openvswitch-2.10-0.1.el7ost.noarch                    6/19 
  Verifying  : openvswitch2.10-test-2.10.0-28.el7fdp.1.noarch              7/19 
  Verifying  : rhosp-openvswitch-ovn-vtep-2.10-0.1.el7ost.noarch           8/19 
  Verifying  : python-twisted-16.1.1-5.1.el7ost.x86_64                     9/19 
  Verifying  : rhosp-openvswitch-devel-2.10-0.1.el7ost.noarch             10/19 
  Verifying  : rhosp-openvswitch-ovn-host-2.10-0.1.el7ost.noarch          11/19 
  Verifying  : 1:openstack-neutron-openvswitch-13.0.2-0.20180929022427.   12/19 
  Verifying  : python-service-identity-14.0.0-4.el7ost.noarch             13/19 
  Verifying  : openvswitch2.10-ovn-vtep-2.10.0-28.el7fdp.1.x86_64         14/19 
  Verifying  : rhosp-openvswitch-ovn-central-2.10-0.1.el7ost.noarch       15/19 
  Verifying  : openvswitch2.10-ovn-common-2.10.0-28.el7fdp.1.x86_64       16/19 
  Verifying  : python-characteristic-14.3.0-4.el7ost.noarch               17/19 
  Verifying  : python-zope-interface-4.0.5-4.el7.x86_64                   18/19 
  Verifying  : openvswitch2.10-devel-2.10.0-28.el7fdp.1.x86_64            19/19 

Installed:
  openstack-neutron-openvswitch.noarch 1:13.0.2-0.20180929022427.c7f970c.el7ost 
  openvswitch2.10-devel.x86_64 0:2.10.0-28.el7fdp.1                             
  openvswitch2.10-ovn-central.x86_64 0:2.10.0-28.el7fdp.1                       
  openvswitch2.10-ovn-common.x86_64 0:2.10.0-28.el7fdp.1                        
  openvswitch2.10-ovn-host.x86_64 0:2.10.0-28.el7fdp.1                          
  openvswitch2.10-ovn-vtep.x86_64 0:2.10.0-28.el7fdp.1                          
  openvswitch2.10-test.noarch 0:2.10.0-28.el7fdp.1                              
  rhosp-openvswitch.noarch 0:2.10-0.1.el7ost                                    
  rhosp-openvswitch-devel.noarch 0:2.10-0.1.el7ost                              
  rhosp-openvswitch-ovn-central.noarch 0:2.10-0.1.el7ost                        
  rhosp-openvswitch-ovn-common.noarch 0:2.10-0.1.el7ost                         
  rhosp-openvswitch-ovn-host.noarch 0:2.10-0.1.el7ost                           
  rhosp-openvswitch-ovn-vtep.noarch 0:2.10-0.1.el7ost                           
  rhosp-openvswitch-test.noarch 0:2.10-0.1.el7ost                               

Dependency Installed:
  pyserial.noarch 0:2.6-6.el7                                                   
  python-characteristic.noarch 0:14.3.0-4.el7ost                                
  python-service-identity.noarch 0:14.0.0-4.el7ost                              
  python-twisted.x86_64 0:16.1.1-5.1.el7ost                                     
  python-zope-interface.x86_64 0:4.0.5-4.el7                                    

Complete!
[root@localhost amer]# 

this package was the problem

Installing : rhosp-openvswitch-devel-2.10-0.1.el7ost.noarch             15/19 
  Installing : rhosp-openvswitch-ovn-vtep-2.10-0.1.el7ost.noarch          16/19 
  Installing : rhosp-openvswitch-ovn-central-2.10-0.1.el7ost.noarch       17/19 
  Installing : rhosp-openvswitch-ovn-host-2.10-0.1.el7ost.noarch          18/19 
  Installing : rhosp-openvswitch-ovn-common-2.10-0.1.el7ost.noarch        19/19 

systemctl enable neutron-server.service neutron-openvswitch-agent.service neutron-dhcp-agent.service neutron-metadata-agent.service neutron-ovs-cleanup.service
systemctl start neutron-server.service neutron-openvswitch-agent.service neutron-dhcp-agent.service neutron-metadata-agent.service neutron-ovs-cleanup.service
systemctl enable neutron-l3-agent.service
systemctl start neutron-l3-agent.service

no problem ... I like Technology :)

I will check the Horizon for changes  now

Comment 17 Amer Hwitat 2019-01-17 20:19:18 UTC
neutron net-create External2 --provider:network_type flat --provider:physical_network eno16777736 --router:external=true --shared
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Unable to establish connection to http://192.168.43.110:9696/v2.0/networks: HTTPConnectionPool(host='192.168.43.110', port=9696): Max retries exceeded with url: /v2.0/networks (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fc4df0ba890>: Failed to establish a new connection: [Errno 111] Connection refused',))

there was a system reboot when I was sleeping

added 

[keystone_authtoken]
auth_uri = http://keystone_ip:5000
auth_url = http://keystone_ip:35357
auth_plugin = password
project_domain_id = default
user_domain_id = default
project_name = services
username = cinder
password = [ccinder password]

in cinder.conf

systemctl restart openstack-nova-api 
systemctl restart openstack-nova-cert 
systemctl restart openstack-nova-consoleauth 
systemctl restart openstack-nova-scheduler 
systemctl restart openstack-nova-conductor 
systemctl restart openstack-nova-novncproxy
systemctl restart neutron-server 
systemctl restart neutron-dhcp-agent
systemctl restart neutron-l3-agent
systemctl restart neutron-metadata-agent
systemctl restart neutron-openvswitch-agent
systemctl restart openstack-cinder-api
systemctl restart openstack-cinder-backup
systemctl restart openstack-cinder-scheduler
systemctl restart openstack-cinder-volume

sudo service --status-all | grep nova
sudo service --status-all | grep neutron
sudo service --status-all | grep cinder

just to make sure

Comment 18 Amer Hwitat 2019-01-19 06:49:37 UTC

I have installed openstack completely without Errors
build the External internal router using CLI and Horizon ... there is an option to create External Networks but inside the Project that you create not on amdin
I have waited because my VM is very slow ... then the router gave an interface with internal network GREEN at last ... the interface with Extrenal network is not it's (RED-DOWN)
then the openstack crashed (Horizon Dashboard) ... then my system became very very slow ...
then I executed some CLI commands to trouble shoot ... the Horizon httpd was down for the dashboard and I learned that I have to install ironic baremetal to clean node because all openstack CLI commannds didn't execute ...
then my laptop crashed again ... this is when I learned that software really crashes hardware and hits very bad :)
then I rebooted and executed

packstack --answer-file=/root/answer.txt --timeout=99999999999999999 --debug #### again I enabled the ironic install to y

then executed the following

export OS_USERNAME=admin
export OS_PASSWORD=server
export OS_TENANT_NAME=admin
export OS_AUTH_URL=http://localhost:5000/v3

neutron net-create External1 --provider:network_type flat --provider:physical_network br-ex --router:external=true --shared
neutron net-create External2 --provider:network_type flat --provider:physical_network eno16777736 --router:external=true --shared
neutron net-create External3 --provider-physical-network provider --provider:physical_network eno16777736 --router:external=true --shared
openstack subnet create --network provider \
  --allocation-pool start=192.168.43.2,end=192.168.43.240 \
  --dns-nameserver 192.168.43.1 --gateway 192.168.43.1 \
  --subnet-range 192.168.43.0/24 provider

mysql
create database neutron;
grant all privileges on neutron.* to 'neutron'@'localhost' identified by 'server';
grant all privileges on neutron.* to 'neutron'@'%' identified by 'server';
quit

export | grep OS_declare -x OS_AUTH_URL="https://192.168.64.128:5000/v3"

#source admin-openrc.sh
. /root/keystonerc_admin
openstack user create --domain default --password-prompt neutron
openstack role add --project service --user neutron admin
openstack service create --name neutron --description "OpenStack Networking" network
openstack endpoint create --region RegionOne network public http://controller:9696
openstack endpoint create --region RegionOne network internal http://controller:9696
openstack endpoint create --region RegionOne network admin http://controller:9696

systemctl enable neutron-server.service neutron-openvswitch-agent.service neutron-dhcp-agent.service neutron-metadata-agent.service neutron-ovs-cleanup.service
systemctl start neutron-server.service neutron-openvswitch-agent.service neutron-dhcp-agent.service neutron-metadata-agent.service neutron-ovs-cleanup.service
systemctl enable neutron-l3-agent.service
systemctl start neutron-l3-agent.service

which is good and bad ... it seems my last configuration was not deleted from the system it gave me the following:

[root@localhost ~]# cat keystonerc_admin
unset OS_SERVICE_TOKEN
    export OS_USERNAME=admin
    export OS_PASSWORD='server'
    export OS_AUTH_URL=http://192.168.43.110:5000/v3
    export PS1='[\u@\h \W(keystone_admin)]\$ '

export OS_PROJECT_NAME=admin
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_DOMAIN_NAME=Default
export OS_IDENTITY_API_VERSION=3
    [root@localhost ~]# . keystoner_admin
-bash: keystoner_admin: No such file or directory
[root@localhost ~]# opestack
bash: opestack: command not found...
[root@localhost ~]# openstack
^C
^CTraceback (most recent call last):
  File "/usr/bin/openstack", line 6, in <module>
    from openstackclient.shell import main
  File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 28, in <module>
    from openstackclient.common import clientmanager
  File "/usr/lib/python2.7/site-packages/openstackclient/common/clientmanager.py", line 191, in <module>
    'openstack.cli.base',
  File "/usr/lib/python2.7/site-packages/openstackclient/common/clientmanager.py", line 153, in get_plugin_modules
    for ep in pkg_resources.iter_entry_points(group):
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 527, in iter_entry_points
    entries = dist.get_entry_map(group)
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2574, in get_entry_map
    self._get_metadata('entry_points.txt'), self
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2340, in parse_map
    for group, lines in data:
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2964, in split_sections
    for line in yield_lines(s):
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2161, in yield_lines
    for ss in strs:
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2503, in _get_metadata
    for line in self.get_metadata_lines(name):
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1429, in get_metadata_lines
    return yield_lines(self.get_metadata(name))
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1421, in get_metadata
    return self._get(self._fn(self.egg_info,name))
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1538, in _get
    return stream.read()
KeyboardInterrupt
[root@localhost ~]# openstack user create --domain default --password-prompt neutron
^CTraceback (most recent call last):
  File "/usr/bin/openstack", line 6, in <module>
    from openstackclient.shell import main
  File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 28, in <module>
    from openstackclient.common import clientmanager
  File "/usr/lib/python2.7/site-packages/openstackclient/common/clientmanager.py", line 191, in <module>
    'openstack.cli.base',
  File "/usr/lib/python2.7/site-packages/openstackclient/common/clientmanager.py", line 153, in get_plugin_modules
    for ep in pkg_resources.iter_entry_points(group):
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 527, in iter_entry_points
    entries = dist.get_entry_map(group)
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2574, in get_entry_map
    self._get_metadata('entry_points.txt'), self
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2340, in parse_map
    for group, lines in data:
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2964, in split_sections
    for line in yield_lines(s):
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2161, in yield_lines
    for ss in strs:
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2503, in _get_metadata
    for line in self.get_metadata_lines(name):
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1429, in get_metadata_lines
    return yield_lines(self.get_metadata(name))
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1421, in get_metadata
    return self._get(self._fn(self.egg_info,name))
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1538, in _get
    return stream.read()
KeyboardInterrupt
[root@localhost ~]# neutron net-create External1 --provider:network_type flat --provider:physical_network br-ex --router:external=true --shared
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Auth plugin requires parameters which were not given: auth_url
[root@localhost ~]# neutron net-create External2 --provider:network_type flat --provider:physical_network eno16777736 --router:external=true --shared
^CTraceback (most recent call last):
  File "/usr/bin/neutron", line 6, in <module>
    from neutronclient.shell import main
  File "/usr/lib/python2.7/site-packages/neutronclient/shell.py", line 30, in <module>
    from keystoneauth1 import session
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 40, in <module>
    import osprofiler.web as osprofiler_web
  File "/usr/lib/python2.7/site-packages/osprofiler/web.py", line 20, in <module>
    from osprofiler import profiler
  File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 27, in <module>
    from osprofiler import notifier
  File "/usr/lib/python2.7/site-packages/osprofiler/notifier.py", line 16, in <module>
    from osprofiler.drivers import base
  File "/usr/lib/python2.7/site-packages/osprofiler/drivers/__init__.py", line 2, in <module>
    from osprofiler.drivers import elasticsearch_driver # noqa
  File "/usr/lib/python2.7/site-packages/osprofiler/drivers/elasticsearch_driver.py", line 18, in <module>
    from oslo_config import cfg
  File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 509, in <module>
    from oslo_config import iniparser
  File "/usr/lib/python2.7/site-packages/oslo_config/iniparser.py", line 16, in <module>
    class ParseError(Exception):
KeyboardInterrupt
[root@localhost ~]# export OS_USERNAME=admin
[root@localhost ~]# export OS_PASSWORD=server
[root@localhost ~]# export OS_TENANT_NAME=admin
[root@localhost ~]# export OS_AUTH_URL=http://localhost:5000/v3
[root@localhost ~]#
[root@localhost ~]#
[root@localhost ~]# neutron net-create External1 --provider:network_type flat --provider:physical_network br-ex --router:external=true --shared
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-06f8141b-a387-44b0-8e3b-1dbd65563232)
[root@localhost ~]# neutron net-create External2 --provider:network_type flat --provider:physical_network eno16777736 --router:external=true --shared
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-09fe0995-c646-4ff3-9dbf-df9593394698)
[root@localhost ~]# neutron net-create External3 --provider-physical-network provider --provider:physical_network eno16777736 --router:external=true --shared
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-21229781-fe7a-47ac-ac38-015cb2b79f23)
[root@localhost ~]# openstack subnet create --network provider \
>   --allocation-pool start=192.168.43.2,end=192.168.43.240 \
>   --dns-nameserver 192.168.43.1 --gateway 192.168.43.1 \
>   --subnet-range 192.168.43.0/24 provider
Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-404b8087-cafa-4a8d-b843-f337fe4a3901)
[root@localhost ~]#
[root@localhost ~]# mysql
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 75028
Server version: 10.1.20-MariaDB MariaDB Server

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> create database neutron;
ERROR 1007 (HY000): Can't create database 'neutron'; database exists
MariaDB [(none)]> grant all privileges on neutron.* to 'neutron'@'localhost' identified by 'server';
Query OK, 0 rows affected (0.90 sec)

MariaDB [(none)]> grant all privileges on neutron.* to 'neutron'@'%' identified by 'server';
Query OK, 0 rows affected (0.05 sec)

MariaDB [(none)]> quit
Bye
[root@localhost ~]#
[root@localhost ~]# export | grep OS_declare -x OS_AUTH_URL="https://192.168.64.128:5000/v3"
grep: OS_AUTH_URL=https://192.168.64.128:5000/v3: No such file or directory
[root@localhost ~]#
[root@localhost ~]# #source admin-openrc.sh
[root@localhost ~]# . /root/keystonerc_admin
[root@localhost ~(keystone_admin)]# openstack user create --domain default --password-prompt neutron

Message from syslogd@localhost at Jan 19 01:13:37 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#2 stuck for 26s! [neutron-server:108188]
User Password:
Repeat User Password:
Conflict occurred attempting to store user - Duplicate entry found with name neutron at domain ID default. (HTTP 409) (Request-ID: req-966b39eb-db15-42ad-ad1b-b225645fd1bb)
[root@localhost ~(keystone_admin)]# ^C
You have new mail in /var/spool/mail/root
[root@localhost ~(keystone_admin)]#
[root@localhost ~(keystone_admin)]# openstack role add --project service --user neutron admin
No project with a name or ID of 'service' exists.
[root@localhost ~(keystone_admin)]# openstack service create --name neutron --description "OpenStack Networking" network
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | OpenStack Networking |
| enabled | True |
| id | 7c09e68b5f4444aeb64d724ff65b0bdb |
| name | neutron |
| type | network |
+-------------+----------------------------------+
[root@localhost ~(keystone_admin)]# openstack endpoint create --region RegionOne network public http://controller:9696
Multiple service matches found for 'network', use an ID to be more specific.
[root@localhost ~(keystone_admin)]# openstack endpoint create --region RegionOne network internal http://controller:9696
Multiple service matches found for 'network', use an ID to be more specific.
[root@localhost ~(keystone_admin)]# openstack endpoint create --region RegionOne network admin http://controller:9696
Multiple service matches found for 'network', use an ID to be more specific.
[root@localhost ~(keystone_admin)]#

[root@localhost ~(keystone_admin)]# openstack baremetal node clean 97effffc1f96444d94143b352e37e2fb
usage: openstack baremetal node clean [-h] [--wait [<time-out>]] --clean-steps
                                      <clean-steps>
                                      <node>
openstack baremetal node clean: error: argument --clean-steps is required
[root@localhost ~(keystone_admin)]# openstack baremetal node clean ironic \
>     --clean-steps '[{"interface": "deploy", "step": "erase_devices"}]'
Error contacting Ironic server: Unable to establish connection to https://192.168.43.110:6385/v1/nodes/ironic/states/provision: HTTPSConnectionPool(host='192.168.43.110', port=6385): Max retries exceeded with url: /v1/nodes/ironic/states/provision (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f1be6fde590>: Failed to establish a new connection: [Errno 111] Connection refused',)). Attempt 6 of 6
Unable to establish connection to https://192.168.43.110:6385/v1/nodes/ironic/states/provision: HTTPSConnectionPool(host='192.168.43.110', port=6385): Max retries exceeded with url: /v1/nodes/ironic/states/provision (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f1be6fde590>: Failed to establish a new connection: [Errno 111] Connection refused',))

well it seems that the cleaner wouldn't clean and I dought that the service is there lets see:

[root@localhost ~(keystone_admin)]# sudo systemctl enable openstack-ironic-conductor
[root@localhost ~(keystone_admin)]# sudo systemctl restart openstack-ironic-conductor
[root@localhost ~(keystone_admin)]# sudo systemctl status openstack-ironic-conductor
● openstack-ironic-conductor.service - OpenStack Ironic Conductor service
   Loaded: loaded (/usr/lib/systemd/system/openstack-ironic-conductor.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2019-01-19 01:35:38 EST; 2s ago
 Main PID: 119599 (ironic-conducto)
    Tasks: 1
   CGroup: /system.slice/openstack-ironic-conductor.service
           └─119599 /usr/bin/python2 /usr/bin/ironic-conductor

Jan 19 01:35:38 localhost.localdomain systemd[1]: Started OpenStack Ironic Co...
Hint: Some lines were ellipsized, use -l to show in full.
[root@localhost ~(keystone_admin)]#

it's running and active then it's not installed ... shixxxxx ... the packstack failed (did it while I was sleeping)

anyway that's what I want to tell you now for now .. I might pull out the original VM and packstack again with ironic installation set to y and have a fresh start and execute the script:

export OS_USERNAME=admin
export OS_PASSWORD=server
export OS_TENANT_NAME=admin
export OS_AUTH_URL=http://localhost:5000/v3

neutron net-create External1 --provider:network_type flat --provider:physical_network br-ex --router:external=true --shared
neutron net-create External2 --provider:network_type flat --provider:physical_network eno16777736 --router:external=true --shared
neutron net-create External3 --provider-physical-network provider --provider:physical_network eno16777736 --router:external=true --shared
openstack subnet create --network provider \
  --allocation-pool start=192.168.43.2,end=192.168.43.240 \
  --dns-nameserver 192.168.43.1 --gateway 192.168.43.1 \
  --subnet-range 192.168.43.0/24 provider

mysql
create database neutron;
grant all privileges on neutron.* to 'neutron'@'localhost' identified by 'server';
grant all privileges on neutron.* to 'neutron'@'%' identified by 'server';
quit

export | grep OS_declare -x OS_AUTH_URL="https://192.168.64.128:5000/v3"

#source admin-openrc.sh
. /root/keystonerc_admin
openstack user create --domain default --password-prompt neutron
openstack role add --project service --user neutron admin
openstack service create --name neutron --description "OpenStack Networking" network
openstack endpoint create --region RegionOne network public http://controller:9696
openstack endpoint create --region RegionOne network internal http://controller:9696
openstack endpoint create --region RegionOne network admin http://controller:9696

systemctl enable neutron-server.service neutron-openvswitch-agent.service neutron-dhcp-agent.service neutron-metadata-agent.service neutron-ovs-cleanup.service
systemctl start neutron-server.service neutron-openvswitch-agent.service neutron-dhcp-agent.service neutron-metadata-agent.service neutron-ovs-cleanup.service
systemctl enable neutron-l3-agent.service
systemctl start neutron-l3-agent.service

which is my collection from the Bugs reports sites for Openstack ..

Comment 19 Amer Hwitat 2019-01-19 08:48:27 UTC
[root@localhost network-scripts]# systemctl status network -l
● network.service - LSB: Bring up/down networking
   Loaded: loaded (/etc/rc.d/init.d/network; bad; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sat 2019-01-19 03:47:01 EST; 21s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 86319 ExecStop=/etc/rc.d/init.d/network stop (code=exited, status=0/SUCCESS)
  Process: 86591 ExecStart=/etc/rc.d/init.d/network start (code=exited, status=1/FAILURE)
    Tasks: 0

Jan 19 03:47:01 localhost.localdomain dhclient[86963]: Please report for this software via the Red Hat Bugzilla site:
Jan 19 03:47:01 localhost.localdomain dhclient[86963]:     http://bugzilla.redhat.com
Jan 19 03:47:01 localhost.localdomain dhclient[86963]: ution.
Jan 19 03:47:01 localhost.localdomain dhclient[86963]: exiting.
Jan 19 03:47:01 localhost.localdomain network[86591]: failed.
Jan 19 03:47:01 localhost.localdomain network[86591]: [FAILED]
Jan 19 03:47:01 localhost.localdomain systemd[1]: network.service: control process exited, code=exited status=1
Jan 19 03:47:01 localhost.localdomain systemd[1]: Failed to start LSB: Bring up/down networking.
Jan 19 03:47:01 localhost.localdomain systemd[1]: Unit network.service entered failed state.
Jan 19 03:47:01 localhost.localdomain systemd[1]: network.service failed.
[root@localhost network-scripts]#

Comment 20 Amer Hwitat 2019-01-19 13:21:05 UTC
anyway it was another file named eno16777736_1 inside /etc/sysconfig/network-scripts/ that made the problem ... sorry it's not a bug report

Comment 21 Amer Hwitat 2019-01-21 10:56:05 UTC
Thanks guys for opening a case for me in RH support cases, and for the survey sent to me I would like to say that the only thing I suffer from now is swift the service is always crashing and becoming down, which is stopping other services like Glance,Nova, neutron, cinder ...etc

and the other thing is that OVS bridge of openvswitch is always giving error and state UNKNOWN to the br-ex I have,read the following:

systemctl restart network #### and check brifge br-ex status with ip a if Down or UNKOWN

grep rewrite /etc/httpd/conf.modules.d/*  ## recreate modules in httpd

systemctl restart httpd
systemctl restart openstack*
systemctl status openstack*
#########################################################################################
gedit /etc/neutron/plugin.ini #### define openvswitch ovs and flat networks

File : openvswitch_agent.ini

integration_bridge=br-int
tenant_network_type = local,flat,vlan
network_vlan_ranges = physnet1
enable_tunneling = True
bridge_mappings = physnet1:br-ex
physical_interface_mappings = physnet1:br-ex

#/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini
[ovs]
#change br-eth0 to your "data network" bridge
bridge_mappings = physnet1:br-eth0
network_vlan_ranges = physnet1

systemctl restart neutron-openvswitch-agent

In addition, turning on verbose and debugging mode in the neutron conf can help alot in tracking down this issue.

#/etc/neutron/neutron.conf
[DEFAULT]
verbose = True
debug = True
#########################################################################################
systemctl enable neutron-server.service neutron-openvswitch-agent.service neutron-dhcp-agent.service neutron-metadata-agent.service neutron-ovs-cleanup.service
systemctl start neutron-server.service neutron-openvswitch-agent.service neutron-dhcp-agent.service neutron-metadata-agent.service neutron-ovs-cleanup.service
systemctl enable neutron-l3-agent.service
systemctl start neutron-l3-agent.service
#########################################################################################
export OS_USERNAME=admin
export OS_PASSWORD=server
export OS_TENANT_NAME=admin
export OS_AUTH_URL=http://192.168.43.110:5000/v3

curl -v http://192.168.43.110:35357
  curl -v http://192.168.43.110:35357


neutron net-create External1 --provider:network_type flat --provider:physical_network extnet --router:external=true --shared
neutron net-create External2 --provider:network_type flat --provider:physical_network extnet --router:external=true --shared
neutron net-create External3 --provider-physical-network provider --provider:physical_network br-ex --router:external=true --shared

openstack subnet create --network provider \
  --allocation-pool start=192.168.43.2,end=192.168.43.240 \
  --dns-nameserver 192.168.43.1 --gateway 192.168.43.1 \
  --subnet-range 192.168.43.0/24 provider

openstack network create --provider-network-type flat \
                         --provider-physical-network extnet \
                         --external \
                         --share \
                         external_network
[root@localhost ~(keystone_admin)]# openstack network create --provider-network-type flat \
>                          --provider-physical-network extnet \
>                          --external \
>                          --share \
>                          external_network
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | UP                                   |
| availability_zone_hints   |                                      |
| availability_zones        |                                      |
| created_at                | 2019-01-20T15:41:16Z                 |
| description               |                                      |
| dns_domain                | None                                 |
| id                        | a69b0d71-5d8b-4d7b-afdc-724235c971ec |
| ipv4_address_scope        | None                                 |
| ipv6_address_scope        | None                                 |
| is_default                | False                                |
| is_vlan_transparent       | None                                 |
| mtu                       | 1500                                 |
| name                      | external_network                     |
| port_security_enabled     | True                                 |
| project_id                | d1d7e8bafc6d4e79a98cb177bcd31325     |
| provider:network_type     | flat                                 |
| provider:physical_network | extnet                               |
| provider:segmentation_id  | None                                 |
| qos_policy_id             | None                                 |
| revision_number           | 1                                    |
| router:external           | External                             |
| segments                  | None                                 |
| shared                    | True                                 |
| status                    | ACTIVE                               |
| subnets                   |                                      |
| tags                      |                                      |
| updated_at                | 2019-01-20T15:41:17Z                 |
+---------------------------+--------------------------------------+
[root@localhost ~(keystone_admin)]#

openstack subnet create --subnet-range 192.168.43.0/24 \
                        --network external_network \
                        --no-dhcp \
                        --gateway=192.168.43.1 \
                        --allocation-pool start=192.168.43.100,end=192.168.43.240 \
                        external_subnet
[root@localhost ~(keystone_admin)]# openstack subnet create --subnet-range 192.168.43.0/24 \
>                         --network external_network \
>                         --no-dhcp \
>                         --gateway=192.168.43.1 \
>                         --allocation-pool start=192.168.43.100,end=192.168.43.240 \
>                         external_subnet
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| allocation_pools  | 192.168.43.100-192.168.43.240        |
| cidr              | 192.168.43.0/24                      |
| created_at        | 2019-01-20T15:42:16Z                 |
| description       |                                      |
| dns_nameservers   |                                      |
| enable_dhcp       | False                                |
| gateway_ip        | 192.168.43.1                         |
| host_routes       |                                      |
| id                | 7660b3d7-10c3-48ed-a5a2-84ea253dfa0f |
| ip_version        | 4                                    |
| ipv6_address_mode | None                                 |
| ipv6_ra_mode      | None                                 |
| name              | external_subnet                      |
| network_id        | a69b0d71-5d8b-4d7b-afdc-724235c971ec |
| project_id        | d1d7e8bafc6d4e79a98cb177bcd31325     |
| revision_number   | 0                                    |
| segment_id        | None                                 |
| service_types     |                                      |
| subnetpool_id     | None                                 |
| tags              |                                      |
| updated_at        | 2019-01-20T15:42:16Z                 |
+-------------------+--------------------------------------+
[root@localhost ~(keystone_admin)]#

###########################################################################################
openstack endpoint create ironic \
  --region RegionOne internal \
  https://192.168.43.110:5000/v3

openstack baremetal node list

openstack baremetal node clean node \
    --clean-steps '[{"interface": "deploy", "step": "erase_devices_metadata"}]'

openstack baremetal node clean node \
    --clean-steps '[{"interface": "deploy", "step": "erase_devices"}]'

openstack network show external_network -f value -c id

or automated cleaning happens if you included this in  /etc/ironic/ironic.conf
[DEFAULT]

[conductor]
automated_clean=true
[deploy]
erase_devices_priority=0
[ilo]
clean_priority_erase_devices=0
[deploy]
erase_devices_iterations=1
##############################################################################
the devices will go to cleaning and out the restart or simply reboot you VM

systemctl disable neutron-server.service neutron-openvswitch-agent.service neutron-dhcp-agent.service neutron-metadata-agent.service neutron-ovs-cleanup.service
systemctl stop neutron-server.service neutron-openvswitch-agent.service neutron-dhcp-agent.service neutron-metadata-agent.service neutron-ovs-cleanup.service
systemctl disable neutron-l3-agent.service
systemctl stop neutron-l3-agent.service
#############
systemctl enable neutron-server.service neutron-openvswitch-agent.service neutron-dhcp-agent.service neutron-metadata-agent.service neutron-ovs-cleanup.service
systemctl start neutron-server.service neutron-openvswitch-agent.service neutron-dhcp-agent.service neutron-metadata-agent.service neutron-ovs-cleanup.service
systemctl enable neutron-l3-agent.service
systemctl start neutron-l3-agent.service
#############

key pair

-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEAwOGaWqBPO4QydSG6/qZ9TMV3zMJLLmKpeRr+AT680TyHlnjY
Mjxrue4c2JtYspuheH0FvoD1qHEg/+8l7AAd3n4APvtYYtEGtMDfanghrJYbvRFe
gNjuTB7NAMNTf/+5OK6ZtXU0UKSfD5vOPzmRAPPZZfHxLVgyvXGNvkv8bW9RkYzM
7z/w1x1inmHChp8b6XLbAwu5kOZshT6XINUdnoJFdHQUjTnBS3OpW1VW06I5RxyQ
tZcqCmYozLJmP/IYX7BqY2m9ShT42tmVN8oNmcrz3IYz42a4wZYfhTxqboXNDMfD
U4vNXlRJTK/jFPGV0o/OT/oeIQxqKvTTGGw21QIDAQABAoIBAAercOXVphDpys/Z
jYP+vtDEsWTU/QRVLAQCzMLj6cIwM/wJxh8JRAdaro72bmp7wcn2vrm6iwotpXBH
5CwkPQvR2tSmTfdv9GJPoRKswDoKpgO4oxdgiOlB+Ejle3aFhqn4V2hdxrzGLwUB
3wuokFmHWMey/TIeZbYew9IjV2BKkTensTCBQIr4IbIcjV9LQdP7oG4Q7eboJ7ng
KgquZkFg1M/FdGg3DJ9w8r27FKQ1I6+sSzH85d+bhqbVVASfETeb70n0REhpNFTu
AL3o2hwxqu9ntFuD4tJlAeyzUfCvOELEWckA5q1MFtfe66Y0hhIoS9vaG0XFzhPr
cmx3AQECgYEA5P/j++LxbReZE961LKIQlPDA0looCiCBoCyvQPshGmLBYexRSR2S
B+gaODNozZDibPjcEGRhyfUYvO09G/ig002qdyfKnA4/Re8dNd+rvL9vrjyh91fK
69WpPA8r+vXeOlTTDXga568uOaARtEJEVaGn6UkcMDC18bRQRvlpbPUCgYEA15+E
+eT4KVmt5NPi3KMSDTwjWzUJBIbJIc6mIpbTjV+Zxc4/2ssRJWXyFL+dtxusW6YT
bKSAowebyY26j3Dcx3W3eHNEJB08YS0gmq/IpzJlku2hdctv9GJpz9fIqTuikYqC
5nLaCjVDH/w2GeL3fTnoSTY4Me3f0Ut3U1mddmECgYEAhZVYEtmh5/d6DoM9wJmR
frCqaoq5DJ9FvoASg597Af9Vu5+1EyeyxfX5RYuHuVKpItRQ2ycTfFIDakIiAa0Z
c+m8ZOHLaRlOg6cmrQSbE/HAXi10deq6rrPmkuhuyWifh/tvWk4Ts88W8fqUGUkG
v8UbRBSn4tynZiW/JofbnfkCgYBwvDWRVjatn/a8UlAlnZj9lHjD4hnB5RrZ8yqH
oSYgn2BNvIVERrHDRObqFDlImE5oaI91WRjspDi+i7yD7LVyD8kKT/HLlAGSqT6H
wep+CGp1BItLhD8XnjDGe4KtgMGBqNLB3EDF340l9v7uT+2n6pPZW2DMMhu4WOWd
jZg3YQKBgQC0Y33oH+7xg4RC/Tv627qX/+++qOEdp5ISf2E/DUxVZ+bq7V1oToSL
+KDNCDB8DihqG99NYZYYiPnVko6reT9pLJMGyc6Wybdm1243VYtT+6c3d5upAwFp
IN/fbW/r7p4rp7crn6RVP551TC9V2DD3Dq1ae59x5GFuNscsVzl1PQ==
-----END RSA PRIVATE KEY-----


solved neurtron and created also a key pair

I have to create a vloume on cinder and container on swift (I manually started the service)

[root@localhost ~(keystone_admin)]# systemctl enable openstack-swift-object.service
Failed to execute operation: Connection timed out
[root@localhost ~(keystone_admin)]# systemctl restart openstack-swift-object.service
[root@localhost ~(keystone_admin)]# systemctl status openstack-swift-object.service
● openstack-swift-object.service - OpenStack Object Storage (swift) - Object Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-swift-object.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-01-20 18:31:58 EST; 17s ago
 Main PID: 55513 (swift-object-se)
    Tasks: 1
   CGroup: /system.slice/openstack-swift-object.service
           └─55513 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf

Jan 20 18:31:58 localhost.localdomain systemd[1]: Started OpenStack Object Storage (swift) - Object Server.
[root@localhost ~(keystone_admin)]#


created

increased RAM of VM by 6016 MB = 6 GB

OVS correct settings:


TYPE=OVSBridge
DEVICE=br-ex
DEVICETYPE=ovs
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
ONBOOT=yes
IPADDR=192.168.43.110
PREFIX=24
GATEWAY=192.168.43.1
DNS=192.168.43.1
BOOTPROTO=static
NM_CONTROLLED=no
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
PROXY_METHOD=none
BROWSER_ONLY=no

HWADDR=00:0C:29:37:19:1F
TYPE=OVSPort
DEVICE=eno16777736
DEVICETYPE=ovs
UUID=c0addf73-98fe-464c-804a-b40110fd6157
OVS_BRIDGE=br-ex
BOOTPROTO=none
ONBOOT=yes
PROMISC=yes
PROXY_METHOD=none
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
NM_CONTROLLED=no
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
NAME=eno16777736
PROXY_METHOD=none
BROWSER_ONLY=no

make sure there is no other garbage files in the dir so that it won't stop network.service for you

but it still gives on OVS error on br-ex

issued systemctl restart network and status network still the same ...

Best regards

this is my last comment .. thanks

Comment 22 Amer Hwitat 2019-01-24 22:50:42 UTC
fixed

https://bugs.launchpad.net/neutron/+bug/1811941
https://access.redhat.com/support/cases/#/case/02293077
https://postimg.cc/gallery/29c2o4ftu/
https://imgur.com/a/X6bEPop

install ironic baremetal

Just edit .conf files of all services in openstack and add timeout=99999999 , for  conn_timeoute ... all timeout variantes

edit wsgi.py 

/usr/lib/python2.7/site-packages/swift/common/wsgi.py

add to the header

import ConfigParser,io

grep rewrite /etc/httpd/conf.modules.d/*  ## recreate modules in httpd

systemctl restart httpd
systemctl restart openstack*

Comment 23 Amer Hwitat 2019-01-25 20:58:14 UTC
again

[root@amer ~(keystone_admin)]# openstack compute service list
+----+------------------+-----------------------+----------+---------+-------+----------------------------+
| ID | Binary | Host | Zone | Status | State | Updated At |
+----+------------------+-----------------------+----------+---------+-------+----------------------------+
| 10 | nova-compute | localhost | nova | enabled | down | 2019-01-19T13:14:06.000000 |
| 18 | nova-compute | localhost.localdomain | nova | enabled | down | 2019-01-25T15:34:10.000000 |
| 24 | nova-scheduler | amer.domain.com | internal | enabled | up | 2019-01-25T16:29:29.000000 |
| 26 | nova-consoleauth | amer.domain.com | internal | enabled | up | 2019-01-25T16:29:34.000000 |
| 27 | nova-conductor | amer.domain.com | internal | enabled | up | 2019-01-25T16:29:35.000000 |
| 29 | nova-compute | amer.domain.com | nova | enabled | up | 2019-01-25T16:29:33.000000 |
+----+------------------+-----------------------+----------+---------+-------+----------------------------+
[root@amer ~(keystone_admin)]# openstack compute service delete 10 18
Failed to delete compute service with ID '10': Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.HostMappingNotFound'> (HTTP 500) (Request-ID: req-576dc490-5ac3-4605-8c7f-c0391527842b)
Failed to delete compute service with ID '18': Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.HostMappingNotFound'> (HTTP 500) (Request-ID: req-544d1089-cc74-40f2-a5b4-2ff7ccd24fa5)
2 of 2 compute services failed to delete.
[root@amer ~(keystone_admin)]#

OK ... again

[root@amer ~(keystone_admin)]# systemctl restart *nova*
[root@amer ~(keystone_admin)]# openstack compute service list
+----+------------------+-----------------------+----------+---------+-------+----------------------------+
| ID | Binary | Host | Zone | Status | State | Updated At |
+----+------------------+-----------------------+----------+---------+-------+----------------------------+
| 10 | nova-compute | localhost | nova | enabled | down | 2019-01-19T13:14:06.000000 |
| 18 | nova-compute | localhost.localdomain | nova | enabled | down | 2019-01-25T15:34:10.000000 |
| 24 | nova-scheduler | amer.domain.com | internal | enabled | up | 2019-01-25T16:34:59.000000 |
| 26 | nova-consoleauth | amer.domain.com | internal | enabled | up | 2019-01-25T16:35:06.000000 |
| 27 | nova-conductor | amer.domain.com | internal | enabled | up | 2019-01-25T16:35:05.000000 |
| 29 | nova-compute | amer.domain.com | nova | enabled | up | 2019-01-25T16:35:00.000000 |
+----+------------------+-----------------------+----------+---------+-------+----------------------------+
[root@amer ~(keystone_admin)]#

trying to create instance

I think it relates to the timeout issue

again



/etc/hostname file overrides the new command hostnamectl set-hostname it should be vise versa , anyway the command also

openstack compute service --disable 

gives failure on certain services with hosts that do not exist anymore like localhost

openstack compute service delete ID 

doesn't delete some services giving the error that you have to report to launchpad.net ...


[root@amer ~(keystone_admin)]# openstack compute service delete 10 18
Failed to delete compute service with ID '10': Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.HostMappingNotFound'> (HTTP 500) (Request-ID: req-576dc490-5ac3-4605-8c7f-c0391527842b)
Failed to delete compute service with ID '18': Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.HostMappingNotFound'> (HTTP 500) (Request-ID: req-544d1089-cc74-40f2-a5b4-2ff7ccd24fa5)
2 of 2 compute services failed to delete.
[root@amer ~(keystone_admin)]#

anyway I will go for fresh installation

maybe someone will try to reply to these errors that I have eventually ... I upgraded Memory to 12 GB on laptop and will use 8GB for the VM ...

Comment 24 Amer Hwitat 2019-01-28 12:15:49 UTC
[root@amer ~]# packstack --answer-file=/root/answer.txt --timeout=99999999 --debug
Welcome to the Packstack setup utility

The installation log file is available at: /var/tmp/packstack/20190126-195724-QZZxCZ/openstack-setup.log

Installing:
Clean Up                                             [ DONE ]
Discovering ip protocol version                      [ DONE ]
Setting up ssh keys                                  [ DONE ]
Preparing servers                                    [ DONE ]
Pre installing Puppet and discovering hosts' details [ DONE ]
Preparing pre-install entries                        [ DONE ]
Setting up CACERT                                    [ DONE ]
Preparing AMQP entries                               [ DONE ]
Preparing MariaDB entries                            [ DONE ]
Fixing Keystone LDAP config parameters to be undef if empty[ DONE ]
Preparing Keystone entries                           [ DONE ]
Preparing Glance entries                             [ DONE ]
Checking if the Cinder server has a cinder-volumes vg[ DONE ]
Preparing Cinder entries                             [ DONE ]
Preparing Ironic entries                             [ DONE ]
Preparing Nova API entries                           [ DONE ]
Creating ssh keys for Nova migration                 [ DONE ]
Gathering ssh host keys for Nova migration           [ DONE ]
Preparing Nova Compute entries                       [ DONE ]
Preparing Nova Scheduler entries                     [ DONE ]
Preparing Nova VNC Proxy entries                     [ DONE ]
Preparing OpenStack Network-related Nova entries     [ DONE ]
Preparing Nova Common entries                        [ DONE ]
Preparing Neutron LBaaS Agent entries                [ DONE ]
Preparing Neutron API entries                        [ DONE ]
Preparing Neutron L3 entries                         [ DONE ]
Preparing Neutron L2 Agent entries                   [ DONE ]
Preparing Neutron DHCP Agent entries                 [ DONE ]
Preparing Neutron Metering Agent entries             [ DONE ]
Checking if NetworkManager is enabled and running    [ DONE ]
Preparing OpenStack Client entries                   [ DONE ]
Preparing Horizon entries                            [ DONE ]
Preparing Swift builder entries                      [ DONE ]
Preparing Swift proxy entries                        [ DONE ]
Preparing Swift storage entries                      [ DONE ]
Preparing Gnocchi entries                            [ DONE ]
Preparing Redis entries                              [ DONE ]
Preparing Ceilometer entries                         [ DONE ]
Preparing Aodh entries                               [ DONE ]
Preparing Puppet manifests                           [ DONE ]
Copying Puppet modules and manifests                 [ DONE ]
Applying 192.168.43.110_controller.pp
192.168.43.110_controller.pp:                        [ DONE ]           
Applying 192.168.43.110_network.pp
192.168.43.110_network.pp:                           [ DONE ]        
Applying 192.168.43.110_compute.pp
192.168.43.110_compute.pp:                           [ DONE ]        
Applying Puppet manifests                            [ DONE ]
Finalizing                                           [ DONE ]

 **** Installation completed successfully ******

Additional information:
 * Time synchronization installation was skipped. Please note that unsynchronized time on server instances might be problem for some OpenStack components.
 * NOTE : A selfsigned CA certificate was generated to be used for ssl, you should still change it do subordinate CA cert. In any case please save the contents of /root/packstackca/.
 * File /root/keystonerc_admin has been created on OpenStack client host 192.168.43.110. To use the command line tools you need to source the file.
 * NOTE : A certificate was generated to be used for ssl, You should change the ssl certificate configured in /etc/httpd/conf.d/ssl.conf on 192.168.43.110 to use a CA signed cert.
 * To access the OpenStack Dashboard browse to https://192.168.43.110/dashboard .
Please, find your login credentials stored in the keystonerc_admin in your home directory.
 * The installation log file is available at: /var/tmp/packstack/20190126-195724-QZZxCZ/openstack-setup.log
 * The generated manifests are available at: /var/tmp/packstack/20190126-195724-QZZxCZ/manifests
 * Note temporary directory /var/tmp/packstack/b218579e3784431bb7df5844175b2569 on host 192.168.43.110 was not deleted for debugging purposes.
[root@amer ~]# 

nova-manage cell_v2 discover_hosts --verbose

openstack compute service list

mysql --max_allowed_packet=32G

!/usr/bin/env python2.7
import time
import mysql.connector

add to object-server.conf

bind_timeout = 999
client_timeout = 999
conn_timeout = 999
node_timeout = 999

add to proxy-server.conf

bind_timeout = 999
client_timeout = 999
conn_timeout = 999
node_timeout = 999


systemctl restart openstack*swift*


[root@amer network-scripts]# 
Message from syslogd@amer at Jan 27 12:46:38 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [nova-api:102738]

systemctl status *rabbit*
systemcrl restart *rabbit*

systemctl restart neutron*
systemctl restart openstack*swift*

Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'oslo_messaging.exceptions.MessageDeliveryFailure'> (HTTP 500) (Request-ID: req-f8946aca-b005-486f-a596-bd7eea52265d)

 cat /var/log/nova/nova-api.log|grep -i error

grep rewrite /etc/httpd/conf.modules.d/*  ## recreate modules in httpd

systemctl restart httpd

tail -f /var/log/nova/nova-api.log

set my.cnf to
max_allowed_packet=64M


mysql
create database neutron;
grant all privileges on neutron.* to 'neutron'@'amer.example.com' identified by 'orient11';
grant all privileges on neutron.* to 'neutron'@'%' identified by 'server';
quit

#########################################################################################
systemctl enable neutron-server.service neutron-openvswitch-agent.service neutron-dhcp-agent.service neutron-metadata-agent.service neutron-ovs-cleanup.service
systemctl start neutron-server.service neutron-openvswitch-agent.service neutron-dhcp-agent.service neutron-metadata-agent.service neutron-ovs-cleanup.service
systemctl enable neutron-l3-agent.service
systemctl start neutron-l3-agent.service
#########################################################################################

[root@amer ~(keystone_admin)]# openstack compute service list
+----+------------------+------------------+----------+---------+-------+----------------------------+
| ID | Binary           | Host             | Zone     | Status  | State | Updated At                 |
+----+------------------+------------------+----------+---------+-------+----------------------------+
|  5 | nova-conductor   | amer             | internal | enabled | down  | 2019-01-28T00:43:20.000000 |
|  7 | nova-scheduler   | amer             | internal | enabled | down  | 2019-01-28T00:43:20.000000 |
| 10 | nova-consoleauth | amer             | internal | enabled | down  | 2019-01-28T00:43:22.000000 |
| 11 | nova-consoleauth | amer.example.com | internal | enabled | up    | 2019-01-28T02:38:55.000000 |
| 12 | nova-scheduler   | amer.example.com | internal | enabled | up    | 2019-01-28T02:38:53.000000 |
| 16 | nova-conductor   | amer.example.com | internal | enabled | up    | 2019-01-28T02:38:53.000000 |
| 18 | nova-compute     | amer.example.com | nova     | enabled | up    | 2019-01-28T02:38:46.000000 |
+----+------------------+------------------+----------+---------+-------+----------------------------+
[root@amer ~(keystone_admin)]#

Message from syslogd@amer at Jan 27 19:26:19 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#5 stuck for 26s! [swapper/5:0]

Message from syslogd@amer at Jan 27 19:26:19 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#1 stuck for 27s! [dmeventd:71548]

Message from syslogd@amer at Jan 27 19:27:30 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [6_scheduler:64928]

Message from syslogd@amer at Jan 27 19:31:25 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [ksoftirqd/5:34]

Message from syslogd@amer at Jan 27 19:32:42 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#3 stuck for 33s! [swift-object-up:11358]

Message from syslogd@amer at Jan 27 19:33:55 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#3 stuck for 24s! [dmeventd:71548]

Message from syslogd@amer at Jan 27 19:34:25 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#2 stuck for 65s! [kworker/2:0:59993]

Message from syslogd@amer at Jan 27 19:37:50 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#2 stuck for 24s! [kworker/u256:3:8447]

Message from syslogd@amer at Jan 27 19:37:50 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [ksoftirqd/5:34]

Message from syslogd@amer at Jan 27 19:37:51 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [systemd:11968]

The CPU has been disabled by the guest operating system. Power off or reset the virtual machine.

########################################################################################


well .. it's the VM compatibility with the new OS .. change frameware

Comment 25 Amer Hwitat 2019-01-28 12:16:52 UTC
sorry firmware ... yu pa ni vrot .. bad English excuse me


Note You need to log in before you can comment on or make changes to this bug.