Bug 1514949

Summary: /etc/os-net-config/config.json is empty
Product: Red Hat OpenStack Reporter: David Hill <dhill>
Component: openstack-tripleo-commonAssignee: Bob Fournier <bfournie>
Status: CLOSED ERRATA QA Contact: Alexander Chuzhoy <sasha>
Severity: medium Docs Contact:
Priority: medium    
Version: 11.0 (Ocata)CC: bfournie, dhill, dsneddon, ebarrera, hbrock, jschluet, jslagle, mburns, mcornea, racedoro, rhel-osp-director-maint, sasha, slinaber
Target Milestone: betaKeywords: Triaged
Target Release: 13.0 (Queens)Flags: bfournie: needinfo? (ebarrera)
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-8.0.2-0.20180414062830.5f869f2.el7ost openstack-tripleo-common-8.6.1-0.20180410165748.4d8ca16.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-27 13:39:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1473267    
Attachments:
Description Flags
screenshot of the config.json none

Description David Hill 2017-11-19 16:37:14 UTC
Description of problem:
/etc/os-net-config/config.json is empty but network is properly configured in RHOSP 11.   I'm able to reproduce this on my side.

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Deploy overcloud
2. cat /etc/os-net-config/config.json after the network is configured
3.

Actual results:
Empty

Expected results:
Full

Additional info:
Big Switch plugin relies on this to properly configure itself

Comment 1 David Hill 2017-11-19 16:57:26 UTC
Shouldn't it be something like this instead? 

--- run-os-net-config.sh.orig	2017-11-19 11:48:40.295720753 -0500
+++ run-os-net-config.sh	2017-11-19 11:51:17.024318363 -0500
@@ -109,7 +109,7 @@
     fi
 }
 
-if [ -n '$network_config' ]; then
+if [ -n "$network_config" ]; then
     if [ -z "${disable_configure_safe_defaults:-''}" ]; then
         trap configure_safe_defaults EXIT
     fi

Comment 2 David Hill 2017-11-19 18:19:36 UTC
Something runs and remove the content of /etc/os-net-config/config.json after the network deployment is completed.   I connected to one of the overcloud node and we can see /etc/os-net-config/config.json is actually properly populated with the network configuration json data and after a while, the file is emptied ...  The above patch didn't solve this.

Comment 3 David Hill 2017-11-19 18:23:04 UTC
Created attachment 1355332 [details]
screenshot of the config.json

Comment 4 David Hill 2017-11-19 19:54:38 UTC
It looks like os-apply-config removes the content of /etc/os-net-config/config.json ...

Comment 5 David Hill 2017-11-19 20:00:06 UTC
And it looks like if we delete the template in /usr/libexec-os-apply-config/templates/etc/os-net-config/config.json, it no longer overwrites our file.   I'm wondering if this is a tripleo issue, os-net-config or os-apply-config issue ... since the behavior changed in RHOSP 11, I guess we can't easily fix this by remove the file from the RPM if the same RPM is needed in RHOSP 10 and that we're using that template.

Comment 8 David Hill 2017-11-20 16:58:29 UTC
One confirmed workaround here is to delete the file using virt-customize:

virt-customize -a overcloud-full.qcow2 --delete /usr/libexec/os-apply-config/templates/etc/os-net-config/config.json

We tested this and it solved the problem we experienced before.

Comment 9 Bob Fournier 2017-11-20 18:40:55 UTC
I assume this is the src location of this file - ./instack-undercloud/elements/undercloud-install/os-apply-config/etc/os-net-config/config.json.

Its not clear what affect there is, if any, of removing this from the undercloud install. Investigating...

Comment 11 Bob Fournier 2017-11-20 19:18:16 UTC
Note also the documentation changes here to describe changes to the os-net-config element - https://review.openstack.org/#/c/521197/.

Changing the target release for this bug as there is a workaround for OSP-11.

Comment 13 Bob Fournier 2018-03-22 20:00:47 UTC
Problem appears to not be fixed, so reopening.

We are seeing /etc/os-net-config/config.json empty and os-net-config being invoked by os-apply-config after the network has been configured:
messages:Mar 22 13:59:09 localhost os-collect-config: [2018/03/22 01:58:36 PM] [INFO] Using config file at: /etc/os-net-config/config.json
messages:Mar 22 13:59:20 localhost os-collect-config: [2018/03/22 01:59:20 PM] [INFO] writing /etc/os-net-config/config.json
messages:Mar 22 13:59:30 localhost os-collect-config: [2018/03/22 01:59:30 PM] [INFO] writing /etc/os-net-config/config.json
messages:Mar 22 13:59:43 localhost os-collect-config: [2018/03/22 01:59:43 PM] [INFO] writing /etc/os-net-config/config.json
messages:Mar 22 13:59:55 localhost os-collect-config: [2018/03/22 01:59:55 PM] [INFO] writing /etc/os-net-config/config.json
messages:Mar 22 14:00:18 localhost os-collect-config: [2018/03/22 02:00:18 PM] [INFO] writing /etc/os-net-config/config.json
messages:Mar 22 14:00:33 localhost os-collect-config: [2018/03/22 02:00:33 PM] [INFO] writing /etc/os-net-config/config.json
messages:Mar 22 14:00:46 localhost os-collect-config: [2018/03/22 02:00:46 PM] [INFO] writing /etc/os-net-config/config.json
messages:Mar 22 14:01:03 localhost os-collect-config: [2018/03/22 02:01:03 PM] [INFO] writing /etc/os-net-config/config.json
messages:Mar 22 14:01:23 localhost os-collect-config: [2018/03/22 02:01:23 PM] [INFO] writing /etc/os-net-config/config.json

Its likely that we need to remove the creation of config.json under libexec here:
https://github.com/openstack/tripleo-common/blob/master/heat_docker_agent/configure_container.sh#L102

Comment 14 Bob Fournier 2018-03-22 23:53:59 UTC
Fix upstream will be backported to stab;e/queens - https://review.openstack.org/#/c/555452/

Comment 19 Alexander Chuzhoy 2018-05-07 20:14:10 UTC
Verified:

Environment:
openstack-tripleo-heat-templates-8.0.2-11.el7ost.noarch
openstack-tripleo-common-8.6.1-6.el7ost.noarch



The mentioned file is not empty on OC node:
[root@overcloud-cmp-leaf0-0 ~]# cat /etc/os-net-config/config.json 
{"network_config": [{"addresses": [{"ip_netmask": "10.37.168.155/26"}], "dns_servers": ["10.16.36.29"], "name": "nic1", "routes": [{"ip_netmask": "169.254.169.254/32", "next_hop": "10.37.168.131"}, {"default": true, "next_hop": "10.37.168.190"}], "type": "interface", "use_dhcp": false}, {"members": [{"name": "nic2", "primary": true, "type": "interface"}, {"addresses": [{"ip_netmask": "172.23.1.15/24"}], "routes": [{"ip_netmask": "172.23.0.0/16", "next_hop": "172.23.1.254"}], "type": "vlan", "vlan_id": 1183}, {"addresses": [{"ip_netmask": "172.25.1.10/24"}], "routes": [{"ip_netmask": "172.25.0.0/16", "next_hop": "172.25.1.254"}], "type": "vlan", "vlan_id": 1185}, {"addresses": [{"ip_netmask": "172.19.1.11/24"}], "routes": [{"ip_netmask": "172.19.0.0/16", "next_hop": "172.19.1.254"}], "type": "vlan", "vlan_id": 1189}], "name": "br-ex", "type": "ovs_bridge", "use_dhcp": false}]}

Comment 20 Eduard Barrera 2018-05-13 15:51:04 UTC
I'm experiencing the same issue on 11, but the workaround deleting a file from the overcloud-full image didn't work. Now there is no file at all instead of one empty file.

I can see this on logs:



May 13 11:44:46 overcloud-ceph-env1-0 os-collect-config[2936]: dib-run-parts Sun May 13 11:44:46 EDT 2018 Running /usr/libexec/os-refresh-config/configure.d/20-os-apply-config
May 13 11:44:46 overcloud-ceph-env1-0 os-collect-config[2936]: [2018/05/13 11:44:46 AM] [INFO] writing /var/run/heat-config/heat-config
May 13 11:44:46 overcloud-ceph-env1-0 os-collect-config[2936]: [2018/05/13 11:44:46 AM] [INFO] writing /etc/os-collect-config.conf
May 13 11:44:46 overcloud-ceph-env1-0 os-collect-config[2936]: [2018/05/13 11:44:46 AM] [INFO] success
May 13 11:44:46 overcloud-ceph-env1-0 os-collect-config[2936]: dib-run-parts Sun May 13 11:44:46 EDT 2018 20-os-apply-config completed
May 13 11:44:46 overcloud-ceph-env1-0 os-collect-config[2936]: dib-run-parts Sun May 13 11:44:46 EDT 2018 Running /usr/libexec/os-refresh-config/configure.d/20-os-net-config
May 13 11:44:46 overcloud-ceph-env1-0 os-collect-config[2936]: ++ os-apply-config --key os_net_config --type raw --key-default ''
May 13 11:44:47 overcloud-ceph-env1-0 os-collect-config[2936]: + NET_CONFIG=
May 13 11:44:47 overcloud-ceph-env1-0 os-collect-config[2936]: + '[' -n '' ']'
May 13 11:44:47 overcloud-ceph-env1-0 os-collect-config[2936]: dib-run-parts Sun May 13 11:44:47 EDT 2018 20-os-net-config completed

Comment 22 Bob Fournier 2018-05-13 18:04:49 UTC
Hi Eduard - there is a separate issue reported in OSP-11 - https://bugzilla.redhat.com/show_bug.cgi?id=1574473.  Is this what you are hitting?
Also are you using old-style nic config files or the new style, i.e. with the run-os-net-config.sh script?

As this bug is targeted for osp-13 we should either track what you are seeing in 1574473 or create a new bug targeted to OSP-11. Thanks.

Comment 24 errata-xmlrpc 2018-06-27 13:39:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086