Description of problem: Juniper contrail-compute deployment failed during heat-stack update.During heat update for compute, heat provision stuck at below place and exited the prompt after some hours with an error "Authentication failed" 2017-06-12 22:18:55 [NetworkConfig]: CREATE_IN_PROGRESS state changed 2017-06-12 22:18:57 [NetIpMap]: CREATE_COMPLETE state changed 2017-06-12 22:18:58 [NetworkConfig]: CREATE_COMPLETE state changed 2017-06-12 22:18:58 [NetworkDeployment]: CREATE_IN_PROGRESS state changed ERROR: Authentication failed: Authentication required ====> This error cam after hours Nova list shows the compute VM is active but heat-admin cloudn't login without password. This issue is seen only on compute. Heat-admin able to login with out password on all other overcloud-vm's except compute. I have tried to redeploy the compute after deleting default nova-key pair as mentioned in RH doc- https://access.redhat.com/solutions/2701051 but it didn't help. [stack@undercloud-new ~]$ nova list +--------------------------------------+------------------------+--------+------------+-------------+---------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------------------------+--------+------------+-------------+---------------------+ | fa9f086e-62c0-4583-a825-e4252c8f1206 | overcloud-compute-0 | ACTIVE | - | Running | ctlplane=192.0.2.7 | | fb3068ad-1774-4d91-be1f-277ef9daed81 | overcloud-controller-0 | ACTIVE | - | Running | ctlplane=192.0.2.21 | | ab139d17-0e06-46e1-9795-7363df98fa36 | overcloud-controller-1 | ACTIVE | - | Running | ctlplane=192.0.2.20 | | 6bd8304f-2d10-4d62-9522-5346fe30853a | overcloud-controller-2 | ACTIVE | - | Running | ctlplane=192.0.2.22 | +--------------------------------------+------------------------+--------+------------+-------------+---------------------+ [stack@undercloud-new ~]$ ssh heat-admin.2.7 heat-admin.2.7's password: === Version-Release number of selected component (if applicable): OSP8- RHEL -7.3 How reproducible: Steps to Reproduce: 1.Create overcloud-vms(openstack), stack deployment using director 2.Updated the contrail neutron plugin on overcloud openstack vm's 3. heat-update for compute vm creation. This failed Actual results: Expected results: Additional info: [stack@undercloud-new ~]$ ./deploy-with-contrail.sh + '[' /home/stack '!=' /home/stack ']' + control_scale=3 + compute_scale=1 + ceph_scale=0 + '[' 0 -eq 3 ']' + echo 'control_scale=3, compute_scale=1, ceph_scale=0' control_scale=3, compute_scale=1, ceph_scale=0 +++ dirname ./deploy-with-contrail.sh ++ cd . ++ pwd + DIR=/home/stack + template_base_dir=/home/stack/templates + ntpserver=10.84.5.100 + openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /home/stack/templates/network-envi ronment.yaml -e /home/stack/templates/neutron-opencontrail.yaml --control-flavor control --compute-flavor compute --ceph-storage-flavor ceph-storage --control-sc ale 3 --compute-scale 1 --ceph-storage-scale 0 --ntp-server 10.84.5.100 --neutron-network-type vxlan --neutron-tunnel-types vxlan !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ! WARNING! WARNING! WARNING! WARNING! WARNING! WARNING! WARNING! WARNING! ! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! This version of director has a security issue (CVE-2017-2637) related to libvirtd based live-migration. A mitigation is available but requires a manual "opt-in" step to enable. Please visit https://access.redhat.com/solutions/3022771 for additional details. A future update will provide the full fix and remove this warning. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ! WARNING! WARNING! WARNING! WARNING! WARNING! WARNING! WARNING! WARNING! ! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 1 nodes with profile compute won't be used for deployment now Configuration has 1 warnings, fix them before proceeding. Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates 2017-06-12 22:14:07 [overcloud]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:14:21 [Networks]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:14:21 [VipConfig]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:14:21 [overcloud-Networks-wvwienvqtcls]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:14:21 [InternalNetwork]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:14:21 [ExternalNetwork]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:14:21 [overcloud-Networks-wvwienvqtcls-InternalNetwork-smjco4ltesgo]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:14:21 [overcloud-VipConfig-xvf6vk2ytc5m]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:14:22 [TenantNetwork]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:14:22 [StorageMgmtNetwork]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:14:22 [ManagementNetwork]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:14:22 [StorageNetwork]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:14:22 [overcloud-Networks-wvwienvqtcls-ManagementNetwork-zbsrr776luos]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:14:22 [overcloud-Networks-wvwienvqtcls-StorageMgmtNetwork-su23plfs7ajn]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:14:22 [overcloud-Networks-wvwienvqtcls-TenantNetwork-amnexgpkuioi]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:14:22 [overcloud-Networks-wvwienvqtcls-ExternalNetwork-7m3xouk3zri7]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:14:23 [overcloud-Networks-wvwienvqtcls-ManagementNetwork-zbsrr776luos]: UPDATE_COMPLETE Stack UPDATE completed successfully 2017-06-12 22:14:23 [overcloud-Networks-wvwienvqtcls-StorageNetwork-a7u5idwfqcpw]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:14:23 [overcloud-VipConfig-xvf6vk2ytc5m]: UPDATE_COMPLETE Stack UPDATE completed successfully 2017-06-12 22:14:24 [VipConfig]: UPDATE_COMPLETE state changed 2017-06-12 22:14:24 [ManagementNetwork]: UPDATE_COMPLETE state changed 2017-06-12 22:14:25 [InternalNetwork]: UPDATE_COMPLETE state changed 2017-06-12 22:14:25 [overcloud-Networks-wvwienvqtcls-InternalNetwork-smjco4ltesgo]: UPDATE_COMPLETE Stack UPDATE completed successfully 2017-06-12 22:14:25 [overcloud-Networks-wvwienvqtcls-TenantNetwork-amnexgpkuioi]: UPDATE_COMPLETE Stack UPDATE completed successfully 2017-06-12 22:14:25 [overcloud-Networks-wvwienvqtcls-ExternalNetwork-7m3xouk3zri7]: UPDATE_COMPLETE Stack UPDATE completed successfully 2017-06-12 22:14:26 [TenantNetwork]: UPDATE_COMPLETE state changed 2017-06-12 22:14:26 [overcloud-Networks-wvwienvqtcls-StorageMgmtNetwork-su23plfs7ajn]: UPDATE_COMPLETE Stack UPDATE completed successfully 2017-06-12 22:14:27 [overcloud-Networks-wvwienvqtcls-StorageNetwork-a7u5idwfqcpw]: UPDATE_COMPLETE Stack UPDATE completed successfully 2017-06-12 22:14:30 [overcloud-ObjectStorage-xnzb2ndabuhj]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:14:30 [overcloud-ObjectStorage-xnzb2ndabuhj]: UPDATE_COMPLETE Stack UPDATE completed successfully 2017-06-12 22:14:30 [overcloud-CephStorage-xwo232hpjocj]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:14:30 [overcloud-CephStorage-xwo232hpjocj]: UPDATE_COMPLETE Stack UPDATE completed successfully 2017-06-12 22:14:31 [overcloud-StorageMgmtVirtualIP-w2mcjsuybb3f]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:14:33 [overcloud-PublicVirtualIP-oomtxpuawsx7]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:14:34 [overcloud-PublicVirtualIP-oomtxpuawsx7]: UPDATE_COMPLETE Stack UPDATE completed successfully 2017-06-12 22:14:34 [ObjectStorage]: UPDATE_COMPLETE state changed 2017-06-12 22:14:35 [StorageMgmtVirtualIP]: UPDATE_COMPLETE state changed 2017-06-12 22:14:35 [CephStorage]: UPDATE_COMPLETE state changed 2017-06-12 22:14:35 [PublicVirtualIP]: UPDATE_COMPLETE state changed 2017-06-12 22:14:35 [InternalApiVirtualIP]: UPDATE_COMPLETE state changed 2017-06-12 22:14:35 [VipMap]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:14:36 [overcloud-VipMap-k34yg7wvarjx]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:14:36 [overcloud-VipMap-k34yg7wvarjx]: UPDATE_COMPLETE Stack UPDATE completed successfully 2017-06-12 22:14:37 [VipMap]: UPDATE_COMPLETE state changed 2017-06-12 22:14:37 [EndpointMap]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:14:37 [overcloud-EndpointMap-caijh3lysiny]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:14:38 [EndpointMap]: UPDATE_COMPLETE state changed 2017-06-12 22:14:38 [overcloud-EndpointMap-caijh3lysiny]: UPDATE_COMPLETE Stack UPDATE completed successfully 2017-06-12 22:14:39 [Compute]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:14:39 [overcloud-Compute-ooavwfewbyjh]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:14:40 [BlockStorage]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:14:40 [Controller]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:14:40 [overcloud-BlockStorage-2rbcndtozpoy]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:14:40 [overcloud-BlockStorage-2rbcndtozpoy]: UPDATE_COMPLETE Stack UPDATE completed successfully 2017-06-12 22:14:40 [0]: CREATE_IN_PROGRESS state changed 2017-06-12 22:14:41 [overcloud-Compute-ooavwfewbyjh-0-bdj2oki5dvst]: CREATE_IN_PROGRESS Stack CREATE started 2017-06-12 22:14:42 [NodeAdminUserData]: CREATE_IN_PROGRESS state changed 2017-06-12 22:14:42 [NovaComputeConfig]: CREATE_IN_PROGRESS state changed 2017-06-12 22:14:42 [UpdateConfig]: CREATE_IN_PROGRESS state changed 2017-06-12 22:14:42 [NodeUserData]: CREATE_IN_PROGRESS state changed 2017-06-12 22:14:43 [overcloud-Controller-qdxyoxrjara4]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:14:43 [2]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:14:43 [NovaComputeConfig]: CREATE_COMPLETE state changed 2017-06-12 22:14:44 [overcloud-Controller-qdxyoxrjara4-2-rfzvpmwzp77b]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:14:44 [UpdateConfig]: CREATE_COMPLETE state changed 2017-06-12 22:14:44 [NodeUserData]: CREATE_COMPLETE state changed 2017-06-12 22:14:44 [NodeAdminUserData]: CREATE_COMPLETE state changed 2017-06-12 22:14:44 [UserData]: CREATE_IN_PROGRESS state changed 2017-06-12 22:14:45 [1]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:14:46 [0]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:14:46 [overcloud-Controller-qdxyoxrjara4-1-vde76tdgpihb]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:14:46 [UserData]: CREATE_COMPLETE state changed 2017-06-12 22:14:46 [NovaCompute]: CREATE_IN_PROGRESS state changed 2017-06-12 22:14:47 [overcloud-Controller-qdxyoxrjara4-0-l4wo2gdoq7ej]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:14:57 [UpdateConfig]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:14:57 [NodeUserData]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:14:57 [NodeAdminUserData]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:14:58 [NodeAdminUserData]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:14:58 [UpdateConfig]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:14:58 [NodeUserData]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:14:59 [NodeUserData]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:14:59 [NodeAdminUserData]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:00 [UpdateConfig]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:00 [UpdateConfig]: UPDATE_COMPLETE state changed 2017-06-12 22:15:01 [UpdateConfig]: UPDATE_COMPLETE state changed 2017-06-12 22:15:01 [NodeUserData]: UPDATE_COMPLETE state changed 2017-06-12 22:15:01 [NodeUserData]: UPDATE_COMPLETE state changed 2017-06-12 22:15:01 [NodeUserData]: UPDATE_COMPLETE state changed 2017-06-12 22:15:02 [NodeAdminUserData]: UPDATE_COMPLETE state changed 2017-06-12 22:15:03 [NodeAdminUserData]: UPDATE_COMPLETE state changed 2017-06-12 22:15:03 [TenantPort]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:04 [InternalApiPort]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:04 [UpdateConfig]: UPDATE_COMPLETE state changed 2017-06-12 22:15:04 [ManagementPort]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:04 [ExternalPort]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:05 [StorageMgmtPort]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:05 [InternalApiPort]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:06 [TenantPort]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:06 [NodeAdminUserData]: UPDATE_COMPLETE state changed 2017-06-12 22:15:06 [StoragePort]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:07 [ExternalPort]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:07 [ExternalPort]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:07 [StorageMgmtPort]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:08 [StoragePort]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:08 [ManagementPort]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:09 [ManagementPort]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:09 [StoragePort]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:09 [TenantPort]: UPDATE_COMPLETE state changed 2017-06-12 22:15:10 [ManagementPort]: UPDATE_COMPLETE state changed 2017-06-12 22:15:11 [TenantPort]: UPDATE_COMPLETE state changed 2017-06-12 22:15:11 [InternalApiPort]: UPDATE_COMPLETE state changed 2017-06-12 22:15:11 [StorageMgmtPort]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:11 [ExternalPort]: UPDATE_COMPLETE state changed 2017-06-12 22:15:11 [InternalApiPort]: UPDATE_COMPLETE state changed 2017-06-12 22:15:12 [StorageMgmtPort]: UPDATE_COMPLETE state changed 2017-06-12 22:15:12 [ManagementPort]: UPDATE_COMPLETE state changed 2017-06-12 22:15:12 [TenantPort]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:12 [InternalApiPort]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:12 [StoragePort]: UPDATE_COMPLETE state changed 2017-06-12 22:15:12 [StorageMgmtPort]: UPDATE_COMPLETE state changed 2017-06-12 22:15:13 [NetworkConfig]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:14 [NetIpSubnetMap]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:14 [ExternalPort]: UPDATE_COMPLETE state changed 2017-06-12 22:15:15 [StoragePort]: UPDATE_COMPLETE state changed 2017-06-12 22:15:15 [ManagementPort]: UPDATE_COMPLETE state changed 2017-06-12 22:15:15 [NetIpMap]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:16 [NetworkConfig]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:16 [StorageMgmtPort]: UPDATE_COMPLETE state changed 2017-06-12 22:15:16 [TenantPort]: UPDATE_COMPLETE state changed 2017-06-12 22:15:17 [NetIpSubnetMap]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:17 [NetIpMap]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:17 [InternalApiPort]: UPDATE_COMPLETE state changed 2017-06-12 22:15:17 [StoragePort]: UPDATE_COMPLETE state changed 2017-06-12 22:15:17 [NetworkConfig]: UPDATE_COMPLETE state changed 2017-06-12 22:15:17 [NetIpSubnetMap]: UPDATE_COMPLETE state changed 2017-06-12 22:15:18 [NetIpMap]: UPDATE_COMPLETE state changed 2017-06-12 22:15:19 [NetworkConfig]: UPDATE_COMPLETE state changed 2017-06-12 22:15:19 [NetworkConfig]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:19 [NodeTLSCAData]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:20 [NetIpSubnetMap]: UPDATE_COMPLETE state changed 2017-06-12 22:15:20 [NetIpMap]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:21 [NetIpMap]: UPDATE_COMPLETE state changed 2017-06-12 22:15:21 [NodeTLSCAData]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:21 [NetIpSubnetMap]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:21 [NodeTLSCAData]: UPDATE_COMPLETE state changed 2017-06-12 22:15:22 [NodeTLSCAData]: UPDATE_COMPLETE state changed 2017-06-12 22:15:22 [NodeTLSData]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:22 [NodeTLSData]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:23 [NetIpMap]: UPDATE_COMPLETE state changed 2017-06-12 22:15:23 [NetIpSubnetMap]: UPDATE_COMPLETE state changed 2017-06-12 22:15:23 [NodeTLSData]: UPDATE_COMPLETE state changed 2017-06-12 22:15:24 [NodeTLSData]: UPDATE_COMPLETE state changed 2017-06-12 22:15:24 [NetworkConfig]: UPDATE_COMPLETE state changed 2017-06-12 22:15:24 [NodeTLSCAData]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:24 [ControllerExtraConfigPre]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:25 [ControllerExtraConfigPre]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:25 [NodeTLSCAData]: UPDATE_COMPLETE state changed 2017-06-12 22:15:25 [NodeTLSData]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:26 [NodeTLSData]: UPDATE_COMPLETE state changed 2017-06-12 22:15:27 [ControllerExtraConfigPre]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:28 [ControllerExtraConfigPre]: UPDATE_COMPLETE state changed 2017-06-12 22:15:29 [ControllerExtraConfigPre]: UPDATE_COMPLETE state changed 2017-06-12 22:15:29 [NodeExtraConfig]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:29 [NodeExtraConfig]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:30 [NodeExtraConfig]: UPDATE_COMPLETE state changed 2017-06-12 22:15:31 [NodeExtraConfig]: UPDATE_COMPLETE state changed 2017-06-12 22:15:32 [ControllerExtraConfigPre]: UPDATE_COMPLETE state changed 2017-06-12 22:15:32 [NodeExtraConfig]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:15:33 [NodeExtraConfig]: UPDATE_COMPLETE state changed 2017-06-12 22:15:42 [overcloud-Controller-qdxyoxrjara4-2-rfzvpmwzp77b]: UPDATE_COMPLETE Stack UPDATE completed successfully 2017-06-12 22:15:43 [overcloud-Controller-qdxyoxrjara4-1-vde76tdgpihb]: UPDATE_COMPLETE Stack UPDATE completed successfully 2017-06-12 22:15:44 [2]: UPDATE_COMPLETE state changed 2017-06-12 22:15:44 [1]: UPDATE_COMPLETE state changed 2017-06-12 22:15:46 [0]: UPDATE_COMPLETE state changed 2017-06-12 22:15:46 [overcloud-Controller-qdxyoxrjara4-0-l4wo2gdoq7ej]: UPDATE_COMPLETE Stack UPDATE completed successfully 2017-06-12 22:15:47 [overcloud-Controller-qdxyoxrjara4]: UPDATE_COMPLETE Stack UPDATE completed successfully 2017-06-12 22:15:48 [Controller]: UPDATE_COMPLETE state changed 2017-06-12 22:15:49 [overcloud-AllNodesValidationConfig-5zdvkigmi36l]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:15:50 [overcloud-VipDeployment-klhrtkishgkt]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:15:51 [overcloud-AllNodesValidationConfig-5zdvkigmi36l]: UPDATE_COMPLETE Stack UPDATE completed successfully 2017-06-12 22:15:52 [overcloud-ControllerClusterDeployment-cytprutcpbpz]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:15:53 [overcloud-VipDeployment-klhrtkishgkt]: UPDATE_COMPLETE Stack UPDATE completed successfully 2017-06-12 22:15:54 [overcloud-ControllerIpListMap-au7ebs2p7olw]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:15:54 [overcloud-ControllerIpListMap-au7ebs2p7olw]: UPDATE_COMPLETE Stack UPDATE completed successfully 2017-06-12 22:15:54 [overcloud-ControllerClusterDeployment-cytprutcpbpz]: UPDATE_COMPLETE Stack UPDATE completed successfully 2017-06-12 22:15:54 [overcloud-ControllerBootstrapNodeConfig-ugzf4jvkvyxg]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:15:56 [overcloud-ControllerBootstrapNodeConfig-ugzf4jvkvyxg]: UPDATE_COMPLETE Stack UPDATE completed successfully 2017-06-12 22:15:58 [overcloud-CephClusterConfig-5ooout4l4rvg]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:15:58 [overcloud-ControllerBootstrapNodeDeployment-nw4ltw7pva2l]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:15:59 [overcloud-CephClusterConfig-5ooout4l4rvg]: UPDATE_COMPLETE Stack UPDATE completed successfully 2017-06-12 22:16:00 [overcloud-ControllerBootstrapNodeDeployment-nw4ltw7pva2l]: UPDATE_COMPLETE Stack UPDATE completed successfully 2017-06-12 22:16:00 [ControllerBootstrapNodeDeployment]: UPDATE_COMPLETE state changed 2017-06-12 22:16:01 [CephClusterConfig]: UPDATE_COMPLETE state changed 2017-06-12 22:16:01 [ObjectStorageSwiftDeployment]: UPDATE_COMPLETE state changed 2017-06-12 22:16:01 [ControllerCephDeployment]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:16:01 [overcloud-ControllerSwiftDeployment-rgf7ih42vd5p]: UPDATE_COMPLETE Stack UPDATE completed successfully 2017-06-12 22:16:02 [CephStorageCephDeployment]: UPDATE_IN_PROGRESS state changed 2017-06-12 22:16:02 [overcloud-ControllerCephDeployment-lcq6w3b2nyzn]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:16:02 [overcloud-CephStorageCephDeployment-ry5w5n4kcmvu]: UPDATE_IN_PROGRESS Stack UPDATE started 2017-06-12 22:16:03 [overcloud-ControllerCephDeployment-lcq6w3b2nyzn]: UPDATE_COMPLETE Stack UPDATE completed successfully 2017-06-12 22:16:03 [overcloud-CephStorageCephDeployment-ry5w5n4kcmvu]: UPDATE_COMPLETE Stack UPDATE completed successfully 2017-06-12 22:16:04 [CephStorageCephDeployment]: UPDATE_COMPLETE state changed 2017-06-12 22:16:04 [ControllerCephDeployment]: UPDATE_COMPLETE state changed 2017-06-12 22:16:04 [ControllerSwiftDeployment]: UPDATE_COMPLETE state changed 2017-06-12 22:18:47 [NovaCompute]: CREATE_COMPLETE state changed 2017-06-12 22:18:47 [UpdateDeployment]: CREATE_IN_PROGRESS state changed 2017-06-12 22:18:47 [StoragePort]: CREATE_IN_PROGRESS state changed 2017-06-12 22:18:48 [StorageMgmtPort]: CREATE_IN_PROGRESS state changed 2017-06-12 22:18:49 [ManagementPort]: CREATE_IN_PROGRESS state changed 2017-06-12 22:18:49 [ExternalPort]: CREATE_IN_PROGRESS state changed 2017-06-12 22:18:50 [TenantPort]: CREATE_IN_PROGRESS state changed 2017-06-12 22:18:50 [InternalApiPort]: CREATE_IN_PROGRESS state changed 2017-06-12 22:18:52 [StoragePort]: CREATE_COMPLETE state changed 2017-06-12 22:18:53 [StorageMgmtPort]: CREATE_COMPLETE state changed 2017-06-12 22:18:53 [ExternalPort]: CREATE_COMPLETE state changed 2017-06-12 22:18:53 [ManagementPort]: CREATE_COMPLETE state changed 2017-06-12 22:18:54 [TenantPort]: CREATE_COMPLETE state changed 2017-06-12 22:18:54 [InternalApiPort]: CREATE_COMPLETE state changed 2017-06-12 22:18:55 [NetIpMap]: CREATE_IN_PROGRESS state changed 2017-06-12 22:18:55 [NetworkConfig]: CREATE_IN_PROGRESS state changed 2017-06-12 22:18:57 [NetIpMap]: CREATE_COMPLETE state changed 2017-06-12 22:18:58 [NetworkConfig]: CREATE_COMPLETE state changed 2017-06-12 22:18:58 [NetworkDeployment]: CREATE_IN_PROGRESS state changed ERROR: Authentication failed: Authentication required [stack@undercloud-new ~]$ === [stack@undercloud-new ~]$ ^C [stack@undercloud-new ~]$ nova list +--------------------------------------+------------------------+--------+------------+-------------+---------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------------------------+--------+------------+-------------+---------------------+ | fa9f086e-62c0-4583-a825-e4252c8f1206 | overcloud-compute-0 | ACTIVE | - | Running | ctlplane=192.0.2.7 | | fb3068ad-1774-4d91-be1f-277ef9daed81 | overcloud-controller-0 | ACTIVE | - | Running | ctlplane=192.0.2.21 | | ab139d17-0e06-46e1-9795-7363df98fa36 | overcloud-controller-1 | ACTIVE | - | Running | ctlplane=192.0.2.20 | | 6bd8304f-2d10-4d62-9522-5346fe30853a | overcloud-controller-2 | ACTIVE | - | Running | ctlplane=192.0.2.22 | +--------------------------------------+------------------------+--------+------------+-------------+---------------------+ [stack@undercloud-new ~]$ ssh heat-admin.2.7 heat-admin.2.7's password: [stack@undercloud-new ~]$ nova show fa9f086e-62c0-4583-a825-e4252c8f1206 +--------------------------------------+-----------------------------------------------------------------------------+ | Property | Value | +--------------------------------------+-----------------------------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | undercloud-new.example.com | | OS-EXT-SRV-ATTR:hypervisor_hostname | fcf30b27-e732-498c-9073-41ce311e941e | | OS-EXT-SRV-ATTR:instance_name | instance-00000068 | | OS-EXT-STS:power_state | 1 | | OS-EXT-STS:task_state | - | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2017-06-12T22:18:46.000000 | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | config_drive | True | | created | 2017-06-12T22:14:48Z | | ctlplane network | 192.0.2.7 | | flavor | compute (68d23066-d5ea-4a4d-a492-8685c446ab2c) | | hostId | 4dd02f4b108881a7f817c37dcaa3040909db0ac7587995587a2e36bc | | id | fa9f086e-62c0-4583-a825-e4252c8f1206 | | image | overcloud-full-compute-with-contrail (3384d6de-75db-4f4f-beae-b4945f7e191d) | | key_name | default | | metadata | {} | | name | overcloud-compute-0 | | os-extended-volumes:volumes_attached | [] | | progress | 0 | | security_groups | default | | status | ACTIVE | | tenant_id | 974678ca48a14fb0b5cc55a32a88f57e | | updated | 2017-06-12T22:18:46Z | | user_id | d64a95c1e4ba4d02b27507e64aadc632 | +--------------------------------------+-----------------------------------------------------------------------------+ [stack@undercloud-new ~]$
Contrail vrouter was not provisioned properly. Ip address is assigned to both vhost0 interface as well as the physical interface (eth0). This case the reachability issue to compute vm . While creating vhost0 interface from physical interface, physical interface configuration file shouldn't include IP address and subnetmask. Also need add static instead of dhcp. [root@overcloud-compute-0 ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 52:54:00:d2:c6:f7 brd ff:ff:ff:ff:ff:ff inet 192.0.2.12/24 brd 192.0.2.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::5054:ff:fed2:c6f7/64 scope link valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 52:54:00:5d:36:5d brd ff:ff:ff:ff:ff:ff inet 10.87.67.65/24 brd 10.87.67.255 scope global eth1 valid_lft forever preferred_lft forever inet6 fe80::5054:ff:fe5d:365d/64 scope link valid_lft forever preferred_lft forever 4: vlan20@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000 link/ether 52:54:00:d2:c6:f7 brd ff:ff:ff:ff:ff:ff inet 172.16.2.17/24 brd 172.16.2.255 scope global vlan20 valid_lft forever preferred_lft forever inet6 fe80::5054:ff:fed2:c6f7/64 scope link valid_lft forever preferred_lft forever 5: vlan30@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000 link/ether 52:54:00:d2:c6:f7 brd ff:ff:ff:ff:ff:ff inet 172.16.1.16/24 brd 172.16.1.255 scope global vlan30 valid_lft forever preferred_lft forever inet6 fe80::5054:ff:fed2:c6f7/64 scope link valid_lft forever preferred_lft forever 6: vlan50@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000 link/ether 52:54:00:d2:c6:f7 brd ff:ff:ff:ff:ff:ff inet 172.16.0.14/24 brd 172.16.0.255 scope global vlan50 valid_lft forever preferred_lft forever inet6 fe80::5054:ff:fed2:c6f7/64 scope link valid_lft forever preferred_lft forever 7: pkt1: <UP,LOWER_UP> mtu 65535 qdisc noqueue state UNKNOWN qlen 1 link/void 1e:8f:0f:61:52:2b brd 00:00:00:00:00:00 8: pkt3: <UP,LOWER_UP> mtu 65535 qdisc noqueue state UNKNOWN qlen 1 link/void 5e:09:5c:22:1d:50 brd 00:00:00:00:00:00 9: pkt2: <UP,LOWER_UP> mtu 65535 qdisc noqueue state UNKNOWN qlen 1 link/void ba:1e:c8:b4:78:a8 brd 00:00:00:00:00:00 10: vhost0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000 link/ether 52:54:00:d2:c6:f7 brd ff:ff:ff:ff:ff:ff inet 192.0.2.12/24 brd 192.0.2.255 scope global vhost0 valid_lft forever preferred_lft forever inet6 fe80::5054:ff:fed2:c6f7/64 scope link valid_lft forever preferred_lft forever ==== # This file is autogenerated by os-net-config DEVICE=eth0 ONBOOT=yes HOTPLUG=no NM_CONTROLLED=no PEERDNS=no BOOTPROTO=static IPADDR=192.0.2.12 ========> this shouldn't be here NETMASK=255.255.255.0 =====> this also need to remove from physical interface ~==== #Contrail vhost0 DEVICE=vhost0 ONBOOT=yes BOOTPROTO=none IPV6INIT=no USERCTL=yes IPADDR=192.0.2.12 NETMASK=255.255.255.0 NM_CONTROLLED=no #NETWORK MANAGER BUG WORKAROUND SUBCHANNELS=1,2,3 GATEWAY=10.87.67.126 DNS1=10.84.5.100 DOMAIN="localdomain"
Does (In reply to shaju from comment #1) > Contrail vrouter was not provisioned properly. Ip address is assigned to > both vhost0 interface as well as the physical interface (eth0). This case > the reachability issue to compute vm . While creating vhost0 interface from > physical interface, physical interface configuration file shouldn't include > IP address and subnetmask. Also need add static instead of dhcp. So, does that mean that you solved your own issue?
(In reply to Sven Anderson from comment #2) > Does (In reply to shaju from comment #1) > > Contrail vrouter was not provisioned properly. Ip address is assigned to > > both vhost0 interface as well as the physical interface (eth0). This case > > the reachability issue to compute vm . While creating vhost0 interface from > > physical interface, physical interface configuration file shouldn't include > > IP address and subnetmask. Also need add static instead of dhcp. > > So, does that mean that you solved your own issue? OSP8 Provisioning of Vrouter is done by RH director. This needs to handle in provisioning code.
(In reply to shaju from comment #3) Hi Please let me know if you need any info. Status shows info needed. Puppet code for osp8 provision needs to handle this change. This is managed by RH. Thanks, Shaju > (In reply to Sven Anderson from comment #2) > > Does (In reply to shaju from comment #1) > > > Contrail vrouter was not provisioned properly. Ip address is assigned to > > > both vhost0 interface as well as the physical interface (eth0). This case > > > the reachability issue to compute vm . While creating vhost0 interface from > > > physical interface, physical interface configuration file shouldn't include > > > IP address and subnetmask. Also need add static instead of dhcp. > > > > So, does that mean that you solved your own issue? > > OSP8 Provisioning of Vrouter is done by RH director. This needs to handle in > provisioning code.
Does something need to be changed in nova code or deployment? Or is this rather a neutron issue? I still don't understand what needs to be fixed.
This seems to be a networking issue. Transferred to DFG:Networking and changed component to THT
Changes needed in osp8 puppet provisioning code, I do not know where exact the code change needed.
Networking DFG triage call - We *think* the proper component is os-net-config, switching to HardProv DFG.
Can you please provide the logs from the deployment, either via sosreport or the logs in /var/log, particularly: /var/log/messages /var/log/neutron/* /var/log/heat/* /var/log/openvswitch/* /var/log/nova/* Also please provide the local template files that were used in deployment: /home/stack/templates/network-environment.yaml /home/stack/templates/neutron-opencontrail.yaml
Hello. Please re-open this bug when you have the requested logs and template files. Please also clarify which changes in puppet code you have in mind. If you don't know for sure - please elaborate on why you think that puppet changes are required. Thanks!
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days