Bug 1734178

Summary: OC deployment fails: container-puppet-nova_libvirt fails to start with 'puppet-user: Error: Evaluation Error: Error while evaluating a Function Call, "" is not a valid IP address'
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: openstack-novaAssignee: nova-maint
Status: CLOSED NOTABUG QA Contact: nova-maint
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 15.0 (Stein)CC: dasmith, eglynn, jhakimra, kchamart, mschuppe, sbauza, sgordon, vromanso
Target Milestone: ---Keywords: Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-07-31 07:22:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Alexander Chuzhoy 2019-07-29 21:11:53 UTC
OC deployment fails: container-puppet-nova_libvirt fails to start with 'puppet-user: Error: Evaluation Error: Error while evaluating a Function Call, "" is not a valid IP address'

Environment:
python3-nova-19.0.2-0.20190724220515.25e222c.el8ost.noarch
openstack-nova-conductor-19.0.2-0.20190724220515.25e222c.el8ost.noarch
openstack-nova-common-19.0.2-0.20190724220515.25e222c.el8ost.noarch
python3-novaclient-13.0.1-0.20190710173535.ef842ca.el8ost.noarch
puppet-nova-14.4.1-0.20190718180428.4c5881a.el8ost.noarch
openstack-tripleo-heat-templates-10.6.1-0.20190722170519.014b20c.el8ost.noarch
python3-tripleoclient-11.5.1-0.20190719020420.bffda01.el8ost.noarch



Steps to reproduce:
Attempt to deploy OC on a setup with spine&leaf.


Result:

OC deployment that has composable role and network, consists of 2 leafs (spine&leaf) fails with error on compute node.
Checking the compute node I see that the container-puppet-nova_libvirt failed to start.

[heat-admin@overcloud-novacompute-0 ~]$ sudo podman ps -a
CONTAINER ID  IMAGE                                                   COMMAND               CREATED      STATUS                  PORTS  NAMES
c650ba909259  10.37.168.131:8787/rhosp15/openstack-nova-compute:15.0  /var/lib/containe...  3 hours ago  Exited (1) 2 hours ago         container-puppet-nova_libvirt


Looking at its logs - I see the following error

<13>Jul 29 18:32:57 puppet-user: Error: Evaluation Error: Error while evaluating a Function Call, "" is not a valid IP address. (file: /etc/puppet/modules/tripleo/manifests/profile/base/nova/migration/target.pp, line: 56, column: 31) on node overcloud-novacompute-0


I see that the IPs were set on the nova node according to the network-environment.yaml:
[heat-admin@overcloud-novacompute-0 ~]$ ip -4 -o a
1: lo    inet 127.0.0.1/8 scope host lo\       valid_lft forever preferred_lft forever
2: eth0    inet 10.37.168.158/26 brd 10.37.168.191 scope global eth0\       valid_lft forever preferred_lft forever
6: vlan1185    inet 172.25.1.114/24 brd 172.25.1.255 scope global vlan1185\       valid_lft forever preferred_lft forever
7: vlan1189    inet 172.19.1.125/24 brd 172.19.1.255 scope global vlan1189\       valid_lft forever preferred_lft forever
8: vlan1183    inet 172.23.1.101/24 brd 172.23.1.255 scope global vlan1183\       valid_lft forever preferred_lft forever

Comment 3 Alexander Chuzhoy 2019-07-30 22:03:13 UTC
Thanks for your comment Martin.

The OC was indeed deployed using the old config from previous releases. Namely had:
    Compute0ExtraConfig:
     neutron::agents::ml2::ovs::local_ip: "%{hiera('tenant0')}"


After adjusting to the following (please let me know if you see something that shouldn't be there) the OC deployment successfully passed:
    Compute0ExtraConfig:
     nova::my_ip: "%{hiera('internal_api0')}"
     cold_migration_ssh_inbound_addr: "%{hiera('internal_api0')}"
     live_migration_ssh_inbound_addr: "%{hiera('internal_api0')}"
     nova::compute::libvirt::vncserver_listen: "%{hiera('internal_api0')}"
     nova::compute::vncserver_proxyclient_address: "%{hiera('internal_api0')}"
     nova::migration::libvirt::live_migration_inbound_addr: "%{hiera('internal_api0')}"
     tripleo::profile::base::database::mysql::client::mysql_client_bind_address: "%{hiera('internal_api0')}"
     ovn::controller::ovn_encap_ip: "%{hiera('tenant0')}"

Otherwise this bug can be closed. 
We already have a bug/comment for misconfigured ComputeExtraconfig here:
https://bugzilla.redhat.com/show_bug.cgi?id=1731260#c6

Comment 4 Martin Schuppert 2019-07-31 07:22:35 UTC
(In reply to Alexander Chuzhoy from comment #3)
> Thanks for your comment Martin.
> 
> The OC was indeed deployed using the old config from previous releases.
> Namely had:
>     Compute0ExtraConfig:
>      neutron::agents::ml2::ovs::local_ip: "%{hiera('tenant0')}"
> 
> 
> After adjusting to the following (please let me know if you see something
> that shouldn't be there) the OC deployment successfully passed:
>     Compute0ExtraConfig:
>      nova::my_ip: "%{hiera('internal_api0')}"
>      cold_migration_ssh_inbound_addr: "%{hiera('internal_api0')}"
>      live_migration_ssh_inbound_addr: "%{hiera('internal_api0')}"
>      nova::compute::libvirt::vncserver_listen: "%{hiera('internal_api0')}"
>      nova::compute::vncserver_proxyclient_address:
> "%{hiera('internal_api0')}"
>      nova::migration::libvirt::live_migration_inbound_addr:
> "%{hiera('internal_api0')}"
>     
> tripleo::profile::base::database::mysql::client::mysql_client_bind_address:
> "%{hiera('internal_api0')}"
>      ovn::controller::ovn_encap_ip: "%{hiera('tenant0')}"
> 
> Otherwise this bug can be closed. 
> We already have a bug/comment for misconfigured ComputeExtraconfig here:
> https://bugzilla.redhat.com/show_bug.cgi?id=1731260#c6

Looks good. I'll close this with the reference to doc bug https://bugzilla.redhat.com/show_bug.cgi?id=1731260#c6