Bug 1461185

Summary: RHEL-7.3-OSP8-Juniper contrail-compute deployment using RH director failed. STUCK after :NetworkDeployment
Product: Red Hat OpenStack Reporter: shaju <shajuvk>
Component: os-net-configAssignee: RHOS Maint <rhos-maint>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Shai Revivo <srevivo>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 8.0 (Liberty)CC: amuller, berrange, bfournie, dasmith, eglynn, gbarros, hbrock, jslagle, kchamart, mburns, rhel-osp-director-maint, sbauza, sferdjao, sgordon, shajuvk, srevivo, svanders, vromanso
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-09-20 17:16:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description shaju 2017-06-13 18:49:07 UTC
Description of problem:
Juniper contrail-compute deployment failed during heat-stack update.During heat update for compute, heat provision stuck at below place and exited the prompt after some hours
with an error "Authentication failed"


2017-06-12 22:18:55 [NetworkConfig]: CREATE_IN_PROGRESS  state changed
2017-06-12 22:18:57 [NetIpMap]: CREATE_COMPLETE  state changed
2017-06-12 22:18:58 [NetworkConfig]: CREATE_COMPLETE  state changed
2017-06-12 22:18:58 [NetworkDeployment]: CREATE_IN_PROGRESS  state changed

ERROR: Authentication failed: Authentication required  ====> This error cam after hours

Nova list shows the compute VM is active but heat-admin cloudn't login without password. This issue is seen only on compute. Heat-admin able to login with out password on all other overcloud-vm's except compute. I have tried to redeploy the compute after deleting default nova-key pair as mentioned in RH doc- https://access.redhat.com/solutions/2701051 but it didn't help.

[stack@undercloud-new ~]$ nova list
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                   | Status | Task State | Power State | Networks            |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| fa9f086e-62c0-4583-a825-e4252c8f1206 | overcloud-compute-0    | ACTIVE | -          | Running     | ctlplane=192.0.2.7  |
| fb3068ad-1774-4d91-be1f-277ef9daed81 | overcloud-controller-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.21 |
| ab139d17-0e06-46e1-9795-7363df98fa36 | overcloud-controller-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.20 |
| 6bd8304f-2d10-4d62-9522-5346fe30853a | overcloud-controller-2 | ACTIVE | -          | Running     | ctlplane=192.0.2.22 |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
[stack@undercloud-new ~]$ ssh heat-admin.2.7
heat-admin.2.7's password:
===


Version-Release number of selected component (if applicable):
OSP8- RHEL -7.3

How reproducible:


Steps to Reproduce:
1.Create overcloud-vms(openstack), stack deployment using director 
2.Updated the contrail neutron plugin on overcloud openstack vm's
3. heat-update for compute vm creation. This failed

Actual results:


Expected results:


Additional info:

[stack@undercloud-new ~]$ ./deploy-with-contrail.sh
+ '[' /home/stack '!=' /home/stack ']'
+ control_scale=3
+ compute_scale=1
+ ceph_scale=0
+ '[' 0 -eq 3 ']'
+ echo 'control_scale=3, compute_scale=1, ceph_scale=0'
control_scale=3, compute_scale=1, ceph_scale=0
+++ dirname ./deploy-with-contrail.sh
++ cd .
++ pwd
+ DIR=/home/stack
+ template_base_dir=/home/stack/templates
+ ntpserver=10.84.5.100
+ openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /home/stack/templates/network-envi
ronment.yaml -e /home/stack/templates/neutron-opencontrail.yaml --control-flavor control --compute-flavor compute --ceph-storage-flavor ceph-storage --control-sc
ale 3 --compute-scale 1 --ceph-storage-scale 0 --ntp-server 10.84.5.100 --neutron-network-type vxlan --neutron-tunnel-types vxlan

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!  WARNING! WARNING! WARNING! WARNING!  WARNING! WARNING! WARNING! WARNING!  !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 This version of director has a security issue (CVE-2017-2637) related to
 libvirtd based live-migration. A mitigation is available but requires a
 manual "opt-in" step to enable.
 Please visit https://access.redhat.com/solutions/3022771 for additional
 details. A future update will provide the full fix and remove this warning.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!  WARNING! WARNING! WARNING! WARNING!  WARNING! WARNING! WARNING! WARNING!  !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

1 nodes with profile compute won't be used for deployment now
Configuration has 1 warnings, fix them before proceeding.
Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates
2017-06-12 22:14:07 [overcloud]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:14:21 [Networks]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:14:21 [VipConfig]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:14:21 [overcloud-Networks-wvwienvqtcls]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:14:21 [InternalNetwork]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:14:21 [ExternalNetwork]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:14:21 [overcloud-Networks-wvwienvqtcls-InternalNetwork-smjco4ltesgo]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:14:21 [overcloud-VipConfig-xvf6vk2ytc5m]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:14:22 [TenantNetwork]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:14:22 [StorageMgmtNetwork]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:14:22 [ManagementNetwork]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:14:22 [StorageNetwork]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:14:22 [overcloud-Networks-wvwienvqtcls-ManagementNetwork-zbsrr776luos]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:14:22 [overcloud-Networks-wvwienvqtcls-StorageMgmtNetwork-su23plfs7ajn]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:14:22 [overcloud-Networks-wvwienvqtcls-TenantNetwork-amnexgpkuioi]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:14:22 [overcloud-Networks-wvwienvqtcls-ExternalNetwork-7m3xouk3zri7]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:14:23 [overcloud-Networks-wvwienvqtcls-ManagementNetwork-zbsrr776luos]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2017-06-12 22:14:23 [overcloud-Networks-wvwienvqtcls-StorageNetwork-a7u5idwfqcpw]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:14:23 [overcloud-VipConfig-xvf6vk2ytc5m]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2017-06-12 22:14:24 [VipConfig]: UPDATE_COMPLETE  state changed
2017-06-12 22:14:24 [ManagementNetwork]: UPDATE_COMPLETE  state changed
2017-06-12 22:14:25 [InternalNetwork]: UPDATE_COMPLETE  state changed
2017-06-12 22:14:25 [overcloud-Networks-wvwienvqtcls-InternalNetwork-smjco4ltesgo]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2017-06-12 22:14:25 [overcloud-Networks-wvwienvqtcls-TenantNetwork-amnexgpkuioi]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2017-06-12 22:14:25 [overcloud-Networks-wvwienvqtcls-ExternalNetwork-7m3xouk3zri7]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2017-06-12 22:14:26 [TenantNetwork]: UPDATE_COMPLETE  state changed
2017-06-12 22:14:26 [overcloud-Networks-wvwienvqtcls-StorageMgmtNetwork-su23plfs7ajn]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2017-06-12 22:14:27 [overcloud-Networks-wvwienvqtcls-StorageNetwork-a7u5idwfqcpw]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2017-06-12 22:14:30 [overcloud-ObjectStorage-xnzb2ndabuhj]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:14:30 [overcloud-ObjectStorage-xnzb2ndabuhj]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2017-06-12 22:14:30 [overcloud-CephStorage-xwo232hpjocj]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:14:30 [overcloud-CephStorage-xwo232hpjocj]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2017-06-12 22:14:31 [overcloud-StorageMgmtVirtualIP-w2mcjsuybb3f]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:14:33 [overcloud-PublicVirtualIP-oomtxpuawsx7]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:14:34 [overcloud-PublicVirtualIP-oomtxpuawsx7]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2017-06-12 22:14:34 [ObjectStorage]: UPDATE_COMPLETE  state changed
2017-06-12 22:14:35 [StorageMgmtVirtualIP]: UPDATE_COMPLETE  state changed
2017-06-12 22:14:35 [CephStorage]: UPDATE_COMPLETE  state changed
2017-06-12 22:14:35 [PublicVirtualIP]: UPDATE_COMPLETE  state changed
2017-06-12 22:14:35 [InternalApiVirtualIP]: UPDATE_COMPLETE  state changed
2017-06-12 22:14:35 [VipMap]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:14:36 [overcloud-VipMap-k34yg7wvarjx]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:14:36 [overcloud-VipMap-k34yg7wvarjx]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2017-06-12 22:14:37 [VipMap]: UPDATE_COMPLETE  state changed
2017-06-12 22:14:37 [EndpointMap]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:14:37 [overcloud-EndpointMap-caijh3lysiny]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:14:38 [EndpointMap]: UPDATE_COMPLETE  state changed
2017-06-12 22:14:38 [overcloud-EndpointMap-caijh3lysiny]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2017-06-12 22:14:39 [Compute]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:14:39 [overcloud-Compute-ooavwfewbyjh]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:14:40 [BlockStorage]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:14:40 [Controller]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:14:40 [overcloud-BlockStorage-2rbcndtozpoy]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:14:40 [overcloud-BlockStorage-2rbcndtozpoy]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2017-06-12 22:14:40 [0]: CREATE_IN_PROGRESS  state changed
2017-06-12 22:14:41 [overcloud-Compute-ooavwfewbyjh-0-bdj2oki5dvst]: CREATE_IN_PROGRESS  Stack CREATE started
2017-06-12 22:14:42 [NodeAdminUserData]: CREATE_IN_PROGRESS  state changed
2017-06-12 22:14:42 [NovaComputeConfig]: CREATE_IN_PROGRESS  state changed
2017-06-12 22:14:42 [UpdateConfig]: CREATE_IN_PROGRESS  state changed
2017-06-12 22:14:42 [NodeUserData]: CREATE_IN_PROGRESS  state changed
2017-06-12 22:14:43 [overcloud-Controller-qdxyoxrjara4]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:14:43 [2]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:14:43 [NovaComputeConfig]: CREATE_COMPLETE  state changed
2017-06-12 22:14:44 [overcloud-Controller-qdxyoxrjara4-2-rfzvpmwzp77b]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:14:44 [UpdateConfig]: CREATE_COMPLETE  state changed
2017-06-12 22:14:44 [NodeUserData]: CREATE_COMPLETE  state changed
2017-06-12 22:14:44 [NodeAdminUserData]: CREATE_COMPLETE  state changed
2017-06-12 22:14:44 [UserData]: CREATE_IN_PROGRESS  state changed
2017-06-12 22:14:45 [1]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:14:46 [0]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:14:46 [overcloud-Controller-qdxyoxrjara4-1-vde76tdgpihb]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:14:46 [UserData]: CREATE_COMPLETE  state changed
2017-06-12 22:14:46 [NovaCompute]: CREATE_IN_PROGRESS  state changed
2017-06-12 22:14:47 [overcloud-Controller-qdxyoxrjara4-0-l4wo2gdoq7ej]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:14:57 [UpdateConfig]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:14:57 [NodeUserData]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:14:57 [NodeAdminUserData]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:14:58 [NodeAdminUserData]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:14:58 [UpdateConfig]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:14:58 [NodeUserData]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:14:59 [NodeUserData]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:14:59 [NodeAdminUserData]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:00 [UpdateConfig]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:00 [UpdateConfig]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:01 [UpdateConfig]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:01 [NodeUserData]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:01 [NodeUserData]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:01 [NodeUserData]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:02 [NodeAdminUserData]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:03 [NodeAdminUserData]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:03 [TenantPort]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:04 [InternalApiPort]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:04 [UpdateConfig]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:04 [ManagementPort]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:04 [ExternalPort]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:05 [StorageMgmtPort]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:05 [InternalApiPort]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:06 [TenantPort]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:06 [NodeAdminUserData]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:06 [StoragePort]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:07 [ExternalPort]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:07 [ExternalPort]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:07 [StorageMgmtPort]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:08 [StoragePort]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:08 [ManagementPort]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:09 [ManagementPort]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:09 [StoragePort]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:09 [TenantPort]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:10 [ManagementPort]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:11 [TenantPort]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:11 [InternalApiPort]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:11 [StorageMgmtPort]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:11 [ExternalPort]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:11 [InternalApiPort]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:12 [StorageMgmtPort]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:12 [ManagementPort]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:12 [TenantPort]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:12 [InternalApiPort]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:12 [StoragePort]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:12 [StorageMgmtPort]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:13 [NetworkConfig]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:14 [NetIpSubnetMap]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:14 [ExternalPort]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:15 [StoragePort]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:15 [ManagementPort]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:15 [NetIpMap]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:16 [NetworkConfig]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:16 [StorageMgmtPort]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:16 [TenantPort]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:17 [NetIpSubnetMap]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:17 [NetIpMap]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:17 [InternalApiPort]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:17 [StoragePort]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:17 [NetworkConfig]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:17 [NetIpSubnetMap]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:18 [NetIpMap]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:19 [NetworkConfig]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:19 [NetworkConfig]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:19 [NodeTLSCAData]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:20 [NetIpSubnetMap]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:20 [NetIpMap]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:21 [NetIpMap]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:21 [NodeTLSCAData]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:21 [NetIpSubnetMap]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:21 [NodeTLSCAData]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:22 [NodeTLSCAData]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:22 [NodeTLSData]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:22 [NodeTLSData]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:23 [NetIpMap]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:23 [NetIpSubnetMap]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:23 [NodeTLSData]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:24 [NodeTLSData]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:24 [NetworkConfig]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:24 [NodeTLSCAData]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:24 [ControllerExtraConfigPre]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:25 [ControllerExtraConfigPre]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:25 [NodeTLSCAData]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:25 [NodeTLSData]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:26 [NodeTLSData]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:27 [ControllerExtraConfigPre]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:28 [ControllerExtraConfigPre]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:29 [ControllerExtraConfigPre]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:29 [NodeExtraConfig]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:29 [NodeExtraConfig]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:30 [NodeExtraConfig]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:31 [NodeExtraConfig]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:32 [ControllerExtraConfigPre]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:32 [NodeExtraConfig]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:15:33 [NodeExtraConfig]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:42 [overcloud-Controller-qdxyoxrjara4-2-rfzvpmwzp77b]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2017-06-12 22:15:43 [overcloud-Controller-qdxyoxrjara4-1-vde76tdgpihb]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2017-06-12 22:15:44 [2]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:44 [1]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:46 [0]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:46 [overcloud-Controller-qdxyoxrjara4-0-l4wo2gdoq7ej]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2017-06-12 22:15:47 [overcloud-Controller-qdxyoxrjara4]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2017-06-12 22:15:48 [Controller]: UPDATE_COMPLETE  state changed
2017-06-12 22:15:49 [overcloud-AllNodesValidationConfig-5zdvkigmi36l]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:15:50 [overcloud-VipDeployment-klhrtkishgkt]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:15:51 [overcloud-AllNodesValidationConfig-5zdvkigmi36l]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2017-06-12 22:15:52 [overcloud-ControllerClusterDeployment-cytprutcpbpz]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:15:53 [overcloud-VipDeployment-klhrtkishgkt]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2017-06-12 22:15:54 [overcloud-ControllerIpListMap-au7ebs2p7olw]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:15:54 [overcloud-ControllerIpListMap-au7ebs2p7olw]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2017-06-12 22:15:54 [overcloud-ControllerClusterDeployment-cytprutcpbpz]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2017-06-12 22:15:54 [overcloud-ControllerBootstrapNodeConfig-ugzf4jvkvyxg]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:15:56 [overcloud-ControllerBootstrapNodeConfig-ugzf4jvkvyxg]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2017-06-12 22:15:58 [overcloud-CephClusterConfig-5ooout4l4rvg]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:15:58 [overcloud-ControllerBootstrapNodeDeployment-nw4ltw7pva2l]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:15:59 [overcloud-CephClusterConfig-5ooout4l4rvg]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2017-06-12 22:16:00 [overcloud-ControllerBootstrapNodeDeployment-nw4ltw7pva2l]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2017-06-12 22:16:00 [ControllerBootstrapNodeDeployment]: UPDATE_COMPLETE  state changed
2017-06-12 22:16:01 [CephClusterConfig]: UPDATE_COMPLETE  state changed
2017-06-12 22:16:01 [ObjectStorageSwiftDeployment]: UPDATE_COMPLETE  state changed
2017-06-12 22:16:01 [ControllerCephDeployment]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:16:01 [overcloud-ControllerSwiftDeployment-rgf7ih42vd5p]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2017-06-12 22:16:02 [CephStorageCephDeployment]: UPDATE_IN_PROGRESS  state changed
2017-06-12 22:16:02 [overcloud-ControllerCephDeployment-lcq6w3b2nyzn]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:16:02 [overcloud-CephStorageCephDeployment-ry5w5n4kcmvu]: UPDATE_IN_PROGRESS  Stack UPDATE started
2017-06-12 22:16:03 [overcloud-ControllerCephDeployment-lcq6w3b2nyzn]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2017-06-12 22:16:03 [overcloud-CephStorageCephDeployment-ry5w5n4kcmvu]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2017-06-12 22:16:04 [CephStorageCephDeployment]: UPDATE_COMPLETE  state changed
2017-06-12 22:16:04 [ControllerCephDeployment]: UPDATE_COMPLETE  state changed
2017-06-12 22:16:04 [ControllerSwiftDeployment]: UPDATE_COMPLETE  state changed
2017-06-12 22:18:47 [NovaCompute]: CREATE_COMPLETE  state changed
2017-06-12 22:18:47 [UpdateDeployment]: CREATE_IN_PROGRESS  state changed
2017-06-12 22:18:47 [StoragePort]: CREATE_IN_PROGRESS  state changed
2017-06-12 22:18:48 [StorageMgmtPort]: CREATE_IN_PROGRESS  state changed
2017-06-12 22:18:49 [ManagementPort]: CREATE_IN_PROGRESS  state changed
2017-06-12 22:18:49 [ExternalPort]: CREATE_IN_PROGRESS  state changed
2017-06-12 22:18:50 [TenantPort]: CREATE_IN_PROGRESS  state changed
2017-06-12 22:18:50 [InternalApiPort]: CREATE_IN_PROGRESS  state changed
2017-06-12 22:18:52 [StoragePort]: CREATE_COMPLETE  state changed
2017-06-12 22:18:53 [StorageMgmtPort]: CREATE_COMPLETE  state changed
2017-06-12 22:18:53 [ExternalPort]: CREATE_COMPLETE  state changed
2017-06-12 22:18:53 [ManagementPort]: CREATE_COMPLETE  state changed
2017-06-12 22:18:54 [TenantPort]: CREATE_COMPLETE  state changed
2017-06-12 22:18:54 [InternalApiPort]: CREATE_COMPLETE  state changed
2017-06-12 22:18:55 [NetIpMap]: CREATE_IN_PROGRESS  state changed
2017-06-12 22:18:55 [NetworkConfig]: CREATE_IN_PROGRESS  state changed
2017-06-12 22:18:57 [NetIpMap]: CREATE_COMPLETE  state changed
2017-06-12 22:18:58 [NetworkConfig]: CREATE_COMPLETE  state changed
2017-06-12 22:18:58 [NetworkDeployment]: CREATE_IN_PROGRESS  state changed


ERROR: Authentication failed: Authentication required
[stack@undercloud-new ~]$ 

===

[stack@undercloud-new ~]$ ^C
[stack@undercloud-new ~]$ nova list
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                   | Status | Task State | Power State | Networks            |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| fa9f086e-62c0-4583-a825-e4252c8f1206 | overcloud-compute-0    | ACTIVE | -          | Running     | ctlplane=192.0.2.7  |
| fb3068ad-1774-4d91-be1f-277ef9daed81 | overcloud-controller-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.21 |
| ab139d17-0e06-46e1-9795-7363df98fa36 | overcloud-controller-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.20 |
| 6bd8304f-2d10-4d62-9522-5346fe30853a | overcloud-controller-2 | ACTIVE | -          | Running     | ctlplane=192.0.2.22 |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
[stack@undercloud-new ~]$ ssh heat-admin.2.7
heat-admin.2.7's password: 

[stack@undercloud-new ~]$ nova show fa9f086e-62c0-4583-a825-e4252c8f1206
+--------------------------------------+-----------------------------------------------------------------------------+
| Property                             | Value                                                                       |
+--------------------------------------+-----------------------------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                                      |
| OS-EXT-AZ:availability_zone          | nova                                                                        |
| OS-EXT-SRV-ATTR:host                 | undercloud-new.example.com                                                  |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | fcf30b27-e732-498c-9073-41ce311e941e                                        |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000068                                                           |
| OS-EXT-STS:power_state               | 1                                                                           |
| OS-EXT-STS:task_state                | -                                                                           |
| OS-EXT-STS:vm_state                  | active                                                                      |
| OS-SRV-USG:launched_at               | 2017-06-12T22:18:46.000000                                                  |
| OS-SRV-USG:terminated_at             | -                                                                           |
| accessIPv4                           |                                                                             |
| accessIPv6                           |                                                                             |
| config_drive                         | True                                                                        |
| created                              | 2017-06-12T22:14:48Z                                                        |
| ctlplane network                     | 192.0.2.7                                                                   |
| flavor                               | compute (68d23066-d5ea-4a4d-a492-8685c446ab2c)                              |
| hostId                               | 4dd02f4b108881a7f817c37dcaa3040909db0ac7587995587a2e36bc                    |
| id                                   | fa9f086e-62c0-4583-a825-e4252c8f1206                                        |
| image                                | overcloud-full-compute-with-contrail (3384d6de-75db-4f4f-beae-b4945f7e191d) |
| key_name                             | default                                                                     |
| metadata                             | {}                                                                          |
| name                                 | overcloud-compute-0                                                         |
| os-extended-volumes:volumes_attached | []                                                                          |
| progress                             | 0                                                                           |
| security_groups                      | default                                                                     |
| status                               | ACTIVE                                                                      |
| tenant_id                            | 974678ca48a14fb0b5cc55a32a88f57e                                            |
| updated                              | 2017-06-12T22:18:46Z                                                        |
| user_id                              | d64a95c1e4ba4d02b27507e64aadc632                                            |
+--------------------------------------+-----------------------------------------------------------------------------+
[stack@undercloud-new ~]$

Comment 1 shaju 2017-06-15 18:46:21 UTC
Contrail vrouter was not provisioned properly. Ip address is assigned to both vhost0 interface as well as the physical interface (eth0). This case the reachability issue to compute vm . While creating vhost0 interface from physical interface, physical interface configuration file shouldn't include IP address and subnetmask. Also need add static instead of dhcp.

[root@overcloud-compute-0 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:d2:c6:f7 brd ff:ff:ff:ff:ff:ff
    inet 192.0.2.12/24 brd 192.0.2.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fed2:c6f7/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:5d:36:5d brd ff:ff:ff:ff:ff:ff
    inet 10.87.67.65/24 brd 10.87.67.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe5d:365d/64 scope link
       valid_lft forever preferred_lft forever
4: vlan20@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 52:54:00:d2:c6:f7 brd ff:ff:ff:ff:ff:ff
    inet 172.16.2.17/24 brd 172.16.2.255 scope global vlan20
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fed2:c6f7/64 scope link
       valid_lft forever preferred_lft forever
5: vlan30@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 52:54:00:d2:c6:f7 brd ff:ff:ff:ff:ff:ff
    inet 172.16.1.16/24 brd 172.16.1.255 scope global vlan30
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fed2:c6f7/64 scope link
       valid_lft forever preferred_lft forever
6: vlan50@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 52:54:00:d2:c6:f7 brd ff:ff:ff:ff:ff:ff
    inet 172.16.0.14/24 brd 172.16.0.255 scope global vlan50
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fed2:c6f7/64 scope link
       valid_lft forever preferred_lft forever
7: pkt1: <UP,LOWER_UP> mtu 65535 qdisc noqueue state UNKNOWN qlen 1
    link/void 1e:8f:0f:61:52:2b brd 00:00:00:00:00:00
8: pkt3: <UP,LOWER_UP> mtu 65535 qdisc noqueue state UNKNOWN qlen 1
    link/void 5e:09:5c:22:1d:50 brd 00:00:00:00:00:00
9: pkt2: <UP,LOWER_UP> mtu 65535 qdisc noqueue state UNKNOWN qlen 1
    link/void ba:1e:c8:b4:78:a8 brd 00:00:00:00:00:00
10: vhost0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 52:54:00:d2:c6:f7 brd ff:ff:ff:ff:ff:ff
    inet 192.0.2.12/24 brd 192.0.2.255 scope global vhost0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fed2:c6f7/64 scope link
       valid_lft forever preferred_lft forever



====

# This file is autogenerated by os-net-config
DEVICE=eth0
ONBOOT=yes
HOTPLUG=no
NM_CONTROLLED=no
PEERDNS=no
BOOTPROTO=static
IPADDR=192.0.2.12  ========> this shouldn't be here
NETMASK=255.255.255.0  =====> this also need to remove from physical interface
~====


#Contrail vhost0
DEVICE=vhost0
ONBOOT=yes
BOOTPROTO=none
IPV6INIT=no
USERCTL=yes
IPADDR=192.0.2.12   
NETMASK=255.255.255.0
NM_CONTROLLED=no
#NETWORK MANAGER BUG WORKAROUND
SUBCHANNELS=1,2,3   
GATEWAY=10.87.67.126
DNS1=10.84.5.100
DOMAIN="localdomain"

Comment 2 Sven Anderson 2017-06-16 14:09:33 UTC
Does (In reply to shaju from comment #1)
> Contrail vrouter was not provisioned properly. Ip address is assigned to
> both vhost0 interface as well as the physical interface (eth0). This case
> the reachability issue to compute vm . While creating vhost0 interface from
> physical interface, physical interface configuration file shouldn't include
> IP address and subnetmask. Also need add static instead of dhcp.

So, does that mean that you solved your own issue?

Comment 3 shaju 2017-06-16 22:10:31 UTC

(In reply to Sven Anderson from comment #2)
> Does (In reply to shaju from comment #1)
> > Contrail vrouter was not provisioned properly. Ip address is assigned to
> > both vhost0 interface as well as the physical interface (eth0). This case
> > the reachability issue to compute vm . While creating vhost0 interface from
> > physical interface, physical interface configuration file shouldn't include
> > IP address and subnetmask. Also need add static instead of dhcp.
> 
> So, does that mean that you solved your own issue?

OSP8 Provisioning of Vrouter is done by RH director. This needs to handle in provisioning code.

Comment 5 shaju 2017-06-21 18:13:40 UTC
(In reply to shaju from comment #3)
Hi Please let me know if you need any info. Status shows info needed. 
Puppet code for osp8 provision needs to handle this change. This is managed by RH.

Thanks,
Shaju
 
> (In reply to Sven Anderson from comment #2)
> > Does (In reply to shaju from comment #1)
> > > Contrail vrouter was not provisioned properly. Ip address is assigned to
> > > both vhost0 interface as well as the physical interface (eth0). This case
> > > the reachability issue to compute vm . While creating vhost0 interface from
> > > physical interface, physical interface configuration file shouldn't include
> > > IP address and subnetmask. Also need add static instead of dhcp.
> > 
> > So, does that mean that you solved your own issue?
> 
> OSP8 Provisioning of Vrouter is done by RH director. This needs to handle in
> provisioning code.

Comment 6 Sven Anderson 2017-06-23 15:37:03 UTC
Does something need to be changed in nova code or deployment? Or is this rather a neutron issue? I still don't understand what needs to be fixed.

Comment 7 Sven Anderson 2017-06-30 14:33:01 UTC
This seems to be a networking issue. Transferred to DFG:Networking and changed component to THT

Comment 8 shaju 2017-07-10 17:55:14 UTC
Changes needed in osp8 puppet provisioning code, I do not know where exact the code change needed.

Comment 9 Assaf Muller 2017-08-30 13:48:57 UTC
Networking DFG triage call - We *think* the proper component is os-net-config, switching to HardProv DFG.

Comment 10 Bob Fournier 2017-08-30 14:32:27 UTC
Can you please provide the logs from the deployment, either via sosreport or the logs in /var/log, particularly:
/var/log/messages
/var/log/neutron/*
/var/log/heat/*
/var/log/openvswitch/*
/var/log/nova/*

Also please provide the local template files that were used in deployment:
/home/stack/templates/network-environment.yaml
/home/stack/templates/neutron-opencontrail.yaml

Comment 11 Dmitry Tantsur 2017-09-20 17:16:25 UTC
Hello. Please re-open this bug when you have the requested logs and template files. Please also clarify which changes in puppet code you have in mind. If you don't know for sure - please elaborate on why you think that puppet changes are required. Thanks!

Comment 12 Red Hat Bugzilla 2023-09-14 03:59:05 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days