Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be unavailable on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1419391 - one of compute node network always deploy failure, and node hostname is not correct as configuration file settings. [NEEDINFO]
Summary: one of compute node network always deploy failure, and node hostname is not c...
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: Angus Thomas
QA Contact: Omri Hochman
URL:
Whiteboard:
Depends On:
Blocks: 1425066
TreeView+ depends on / blocked
 
Reported: 2017-02-06 02:45 UTC by Bill Jiang
Modified: 2017-10-02 15:25 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
hyper-converged deployment, 1 controller + 2 compute.
Last Closed: 2017-10-02 15:25:49 UTC
Target Upstream Version:
dtantsur: needinfo? (jiangxt6)

Attachments (Terms of Use)
configuration file and journal system logs (36.56 KB, application/zip)
2017-02-06 02:45 UTC, Bill Jiang 		no flags Details
Add an attachment (proposed patch, testcase, etc.)

Description Bill Jiang 2017-02-06 02:45:41 UTC 
Created attachment 1247926 [details]
configuration file and journal system logs

Description of problem:
Network physical connectivity is fine,one of compute node network always deploy failure, and node hostname is not correct as configuration file settings.

Version-Release number of selected component (if applicable):
osp10

How reproducible:
always

Steps to Reproduce:
[stack@director183 ~]$ vi undercloud.conf
[stack@director183 ~]$ sudo yum remove firewalld-0.4.3.2-8.1.el7_3.noarch(manally replace the command: /bin/rpm -e firewalld-0.4.3.2-8.1.el7_3.noarch)
[stack@director183 ~]$ openstack undercloud install
[stack@director183 ~]$ source ~/stackrc
[stack@director183 ~]$ sudo yum install rhosp-director-images rhosp-director-images-ipa
[stack@director183 ~]$ for i in /usr/share/rhosp-director-images/overcloud-full-latest-10.0.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-10.0.tar; do tar -xvf $i; done
[stack@director183 ~]$ openstack overcloud image upload --image-path /home/stack/images/
[stack@director183 ~]$ vi instackenv.json
[stack@director183 ~]$ openstack baremetal import --json ~/instackenv.json
[stack@director183 ~]$ openstack baremetal configure boot
[stack@director183 ~]$ for node in $(openstack baremetal node list --fields uuid -f value) ; do openstack baremetal node manage $node ; done
[stack@director183 ~]$ openstack overcloud node introspect --all-manageable --provide

[stack@director183 ~]$ ironic node-list 
+--------------------------------------+------------+---------------+-------------+--------------------+-------------+
| UUID                                 | Name       | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+------------+---------------+-------------+--------------------+-------------+
| 7e5967c9-9344-4dee-8b03-27b440b5e193 | con_imm185 | None          | power off   | available          | False       |
| c5b51785-6814-4a1e-8efe-f0d1dc7825a0 | com_imm191 | None          | power off   | available          | False       |
| a1a1a57e-f29f-4a0c-bc2b-a0e2486560db | com_imm193 | None          | power off   | available          | False       |
+--------------------------------------+------------+---------------+-------------+--------------------+-------------+

[stack@director183 ~]$ openstack baremetal node set --property capabilities='profile:control,boot_option:local' 7e5967c9-9344-4dee-8b03-27b440b5e193
[stack@director183 ~]$ openstack baremetal node set --property capabilities='profile:compute,boot_option:local' c5b51785-6814-4a1e-8efe-f0d1dc7825a0
[stack@director183 ~]$ openstack baremetal node set --property capabilities='profile:compute,boot_option:local' a1a1a57e-f29f-4a0c-bc2b-a0e2486560db

[stack@director183 ~]$ openstack baremetal configure boot --root-device=smallest

[stack@director183 ~]$ ironic node-update 7e5967c9-9344-4dee-8b03-27b440b5e193 replace properties/capabilities='node:controller-0,boot_option:local'
[stack@director183 ~]$ ironic node-update c5b51785-6814-4a1e-8efe-f0d1dc7825a0 replace properties/capabilities='node:compute-0,boot_option:local'
[stack@director183 ~]$ ironic node-update a1a1a57e-f29f-4a0c-bc2b-a0e2486560db replace properties/capabilities='node:compute-1,boot_option:local'

[stack@director183 ~]$ openstack overcloud deploy --templates -e templates/hyperconverged_extra_ceph_disk/network-isolation.yaml  -e templates/hyperconverged_extra_ceph_disk/network-environment.yaml -e templates/hyperconverged_extra_ceph_disk/storage-environment.yaml -e templates/hyperconverged_extra_ceph_disk/scheduler_hints_env.yaml -e ~/templates/firstboot/userdata.yaml -r templates/hyperconverged_extra_ceph_disk/roles_data.yaml  --control-flavor baremetal --compute-flavor baremetal  --ntp-server 202.118.1.81


Actual results:
[stack@director183 ~]$ heat stack-list
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
+--------------------------------------+------------+---------------+----------------------+--------------+
| id                                   | stack_name | stack_status  | creation_time        | updated_time |
+--------------------------------------+------------+---------------+----------------------+--------------+
| f2d7a58c-f8ce-4534-9945-08fd060ac5f3 | overcloud  | CREATE_FAILED | 2017-01-16T11:14:41Z | None         |
+--------------------------------------+------------+---------------+----------------------+--------------+
[stack@director183 ~]$ heat resource-list overcloud
WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
+-------------------------------------------+----------------------------------------------+-------------------------------------------------+-----------------+----------------------+
| resource_name                             | physical_resource_id                         | resource_type                                   | resource_status | updated_time         |
+-------------------------------------------+----------------------------------------------+-------------------------------------------------+-----------------+----------------------+
| AllNodesDeploySteps                       |                                              | OS::TripleO::PostDeploySteps                    | INIT_COMPLETE   | 2017-01-16T11:14:41Z |
| AllNodesExtraConfig                       |                                              | OS::TripleO::AllNodesExtraConfig                | INIT_COMPLETE   | 2017-01-16T11:14:41Z |
| AllNodesValidationConfig                  | ddf506c1-4941-44ba-84a5-c0dd3459c863         | OS::TripleO::AllNodes::Validation               | CREATE_COMPLETE | 2017-01-16T11:14:41Z |
| BlockStorage                              | c55a043a-7540-4599-867d-ea75e2fc0bc4         | OS::Heat::ResourceGroup                         | CREATE_COMPLETE | 2017-01-16T11:14:41Z |
| BlockStorageAllNodesDeployment            |                                              | OS::Heat::StructuredDeployments                 | INIT_COMPLETE   | 2017-01-16T11:14:41Z |
| BlockStorageAllNodesValidationDeployment  |                                              | OS::Heat::StructuredDeployments                 | INIT_COMPLETE   | 2017-01-16T11:14:41Z |
| BlockStorageHostsDeployment               |                                              | OS::Heat::StructuredDeployments                 | INIT_COMPLETE   | 2017-01-16T11:14:41Z |
| BlockStorageIpListMap                     | f275665c-84d3-4e22-952f-04f418bbd5e6         | OS::TripleO::Network::Ports::NetIpListMap       | CREATE_COMPLETE | 2017-01-16T11:14:41Z |
| BlockStorageServiceChain                  | e866e2e8-6dff-4d16-a5c6-e474ca817fb0         | OS::TripleO::Services                           | CREATE_COMPLETE | 2017-01-16T11:14:41Z |
| Compute                                   | 78af026c-5053-4162-a71f-26c6457a380f         | OS::Heat::ResourceGroup                         | CREATE_FAILED   | 2017-01-16T11:14:41Z |
| ComputeAllNodesDeployment                 |                                              | OS::Heat::StructuredDeployments                 | INIT_COMPLETE   | 2017-01-16T11:14:41Z |
| ComputeAllNodesValidationDeployment       |                                              | OS::Heat::StructuredDeployments                 | INIT_COMPLETE   | 2017-01-16T11:14:41Z |
| ComputeHostsDeployment                    |                                              | OS::Heat::StructuredDeployments                 | INIT_COMPLETE   | 2017-01-16T11:14:41Z |
| ComputeIpListMap                          |                                              | OS::TripleO::Network::Ports::NetIpListMap       | INIT_COMPLETE   | 2017-01-16T11:14:41Z |
| ComputeServiceChain                       | 24ffb656-b053-4546-b273-768f345e7fe5         | OS::TripleO::Services                           | CREATE_COMPLETE | 2017-01-16T11:14:41Z |
| ControlVirtualIP                          | d3bba820-9532-482d-be58-f95da3125500         | OS::Neutron::Port                               | CREATE_COMPLETE | 2017-01-16T11:14:41Z |
| Controller                                | 54ab1712-22b3-4fe4-b415-e0b6c88e8e00         | OS::Heat::ResourceGroup                         | CREATE_COMPLETE | 2017-01-16T11:14:41Z |
| ControllerAllNodesDeployment              |                                              | OS::Heat::StructuredDeployments                 | INIT_COMPLETE   | 2017-01-16T11:14:41Z |
| ControllerAllNodesValidationDeployment    |                                              | OS::Heat::StructuredDeployments                 | INIT_COMPLETE   | 2017-01-16T11:14:41Z |
| ControllerHostsDeployment                 |                                              | OS::Heat::StructuredDeployments                 | INIT_COMPLETE   | 2017-01-16T11:14:41Z |
| ControllerIpListMap                       | 0258dea0-31b5-426e-b3cb-984351dbbae9         | OS::TripleO::Network::Ports::NetIpListMap       | CREATE_COMPLETE | 2017-01-16T11:14:41Z |
| ControllerServiceChain                    | ba495085-c06c-4c45-a02b-48cf5831b650         | OS::TripleO::Services                           | CREATE_COMPLETE | 2017-01-16T11:14:41Z |
| DefaultPasswords                          | 9237c1d9-2979-423f-b66f-f436216191d4         | OS::TripleO::DefaultPasswords                   | CREATE_COMPLETE | 2017-01-16T11:14:41Z |
| EndpointMap                               | 1ee16862-b514-457b-b659-0fe4aceb185d         | OS::TripleO::EndpointMap                        | CREATE_COMPLETE | 2017-01-16T11:14:41Z |
| HeatAuthEncryptionKey                     | overcloud-HeatAuthEncryptionKey-mwxbufutqafj | OS::Heat::RandomString                          | CREATE_COMPLETE | 2017-01-16T11:14:41Z |
| HorizonSecret                             | overcloud-HorizonSecret-hacih332jz3b         | OS::Heat::RandomString                          | CREATE_COMPLETE | 2017-01-16T11:14:41Z |
| InternalApiVirtualIP                      | 8948a7d4-91db-45dc-95ac-5c0b7e408590         | OS::TripleO::Network::Ports::InternalApiVipPort | CREATE_COMPLETE | 2017-01-16T11:14:41Z |
| MysqlRootPassword                         | overcloud-MysqlRootPassword-cukf4ruj34rj     | OS::Heat::RandomString                          | CREATE_COMPLETE | 2017-01-16T11:14:41Z |
| Networks                                  | 4b8eaa06-bd95-4788-96e2-cdd5d81845f6         | OS::TripleO::Network                            | CREATE_COMPLETE | 2017-01-16T11:14:41Z |
| ObjectStorage                             | fb5847ce-ee79-4af6-b85e-4e941cc0d9f8         | OS::Heat::ResourceGroup                         | CREATE_COMPLETE | 2017-01-16T11:14:41Z |
| ObjectStorageAllNodesDeployment           |                                              | OS::Heat::StructuredDeployments                 | INIT_COMPLETE   | 2017-01-16T11:14:41Z |
| ObjectStorageAllNodesValidationDeployment |                                              | OS::Heat::StructuredDeployments                 | INIT_COMPLETE   | 2017-01-16T11:14:41Z |
| ObjectStorageHostsDeployment              |                                              | OS::Heat::StructuredDeployments                 | INIT_COMPLETE   | 2017-01-16T11:14:41Z |
| ObjectStorageIpListMap                    | b1039e6e-4469-4d7a-aebc-1cec8a95390b         | OS::TripleO::Network::Ports::NetIpListMap       | CREATE_COMPLETE | 2017-01-16T11:14:41Z |
| ObjectStorageServiceChain                 | 909bb7f0-7ed6-4870-97b9-b078cd506015         | OS::TripleO::Services                           | CREATE_COMPLETE | 2017-01-16T11:14:41Z |
| PcsdPassword                              | overcloud-PcsdPassword-b73ittytwer6          | OS::Heat::RandomString                          | CREATE_COMPLETE | 2017-01-16T11:14:41Z |
| PublicVirtualIP                           | 428672f7-2e5e-4561-88e1-862ef592f10e         | OS::TripleO::Network::Ports::ExternalVipPort    | CREATE_COMPLETE | 2017-01-16T11:14:41Z |
| RabbitCookie                              | overcloud-RabbitCookie-5iv5baxxqcza          | OS::Heat::RandomString                          | CREATE_COMPLETE | 2017-01-16T11:14:41Z |
| RedisVirtualIP                            | 7bb165a2-278c-4023-b23a-3814cb551e56         | OS::TripleO::Network::Ports::RedisVipPort       | CREATE_COMPLETE | 2017-01-16T11:14:41Z |
| ServiceNetMap                             | 38b59de3-ce08-4a32-92c5-44135afa6a8b         | OS::TripleO::ServiceNetMap                      | CREATE_COMPLETE | 2017-01-16T11:14:41Z |
| StorageMgmtVirtualIP                      | c9ac348a-145a-4e03-901a-0d837104fdbf         | OS::TripleO::Network::Ports::StorageMgmtVipPort | CREATE_COMPLETE | 2017-01-16T11:14:41Z |
| StorageVirtualIP                          | a2d3ed8e-c971-41d5-9903-73b789825001         | OS::TripleO::Network::Ports::StorageVipPort     | CREATE_COMPLETE | 2017-01-16T11:14:41Z |
| UpdateWorkflow                            |                                              | OS::TripleO::Tasks::UpdateWorkflow              | INIT_COMPLETE   | 2017-01-16T11:14:41Z |
| VipMap                                    | cf6acb98-f0f9-4207-938c-6d8363176202         | OS::TripleO::Network::Ports::NetVipMap          | CREATE_COMPLETE | 2017-01-16T11:14:41Z |
| allNodesConfig                            |                                              | OS::TripleO::AllNodes::SoftwareConfig           | INIT_COMPLETE   | 2017-01-16T11:14:41Z |
| hostsConfig                               |                                              | OS::TripleO::Hosts::SoftwareConfig              | INIT_COMPLETE   | 2017-01-16T11:14:41Z |
+-------------------------------------------+----------------------------------------------+-------------------------------------------------+-----------------+----------------------+

in each host, hostname was set as:
[root@overcloud-controller-pbandark-0 ~]#
[root@overcloud-novacompute-0 ~]#
[root@overcloud-compute-pbandark-0 ~]#

Expected results:
1. all compute nodes deployed successfully
2. all compute nodes'hostname configured correctly.
[root@overcloud-controller-pbandark-0 ~]#
[root@overcloud-compute-pbandark-1 ~]#
[root@overcloud-compute-pbandark-0 ~]#


Additional info:
Comment 1 Red Hat Bugzilla Rules Engine 2017-02-06 02:45:48 UTC 
This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.
Comment 3 Dmitry Tantsur 2017-04-25 10:11:53 UTC 
Hi! Please provide full logs from /var/log or better a sosreport. Does output of 'ironic node-list' show any nodes in any failed state? The same about 'nova list'?
Comment 4 Bob Fournier 2017-10-02 15:25:49 UTC 
As its been over 5 months since logs have been requested, closing this for now. Please reopen if problem can be duplicated and supply the logs.
Note You need to log in before you can comment on or make changes to this bug.